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Preface 


The  Neal-Montgomery  NLP  Evaluation  Methodology  was  developed  under  the  Bench¬ 
mark  Investigation/Identification  Program  as  a  means  of  determining  the  linguistic 
competence  of  Natural  Language  Processing  (NLP)  systems.  Embodied  in  an  evaluation 
tool  based  on  a  detailed  classification  of  linguistic  phenomena  with,  currently,  over  350  test 
items,  the  methodology  produces  descriptive  profiles  of  NLP  system  linguistic  capabilities, 
and  can  be  applied  without  regard  to  the  current  system  application,  domain,  or  specific 
language  processing  task.  Evaluation  data  and  scores  are  explicit  and  thus  open  to  NLP 
community  review  and  critique. 

The  Neal-Montgomery  NLP  Evaluation  Methodology  is  intended  for  the  evaluation  of 
whole  systems  as  opposed  to  the  examination  of  individual  system  components.  To  date,  it 
has  been  applied  to  three  NL  database  query  systems  and  three  text  processing  systems. 

The  evaluation  methodology  is  not  presented  here  as  a  product  to  be  accepted,  in  toto, 
by  the  NLP  community  as  the  standard  for  system  evaluation,  but  rather  as  a  basis  for 
discussion,  critique,  and  possible  refinement  towards  standards  development.  Part  of  that 
evolutionary  process  may  become  concerned,  beyond  the  specific  content  of  the  evaluation 
methodology,  with  the  interpretation  of  evaluation  results.  For  example,  total  consistency 
even  among  multiple  evaluations  of  a  single  system  is  not  practical  and  cannot  be  expected. 
Evaluation  results  obtained  when  an  NLP  system  developer  applies  the  Neal-Montgomery 
NLP  Evaluation  Methodology  within  its  original  domain  may  be  optimal  scores.  Other 
evaluators,  or  evaluations  of  the  same  system  applied  to  new  domains,  could  expect  to  come 
within  a  certain  range  of  those  optimal  scores,  most  likely  without  surpassing  them. 

Researchers  are  invited  to  apply  the  evaluation  methodology  (included  within  this 
report)  to  their  systems  and  report  their  critique  and  recommendations  towards  the  evolution 
of  standard  evaluation  procedures  for  NLP  systems.  Comments  can  also  be  made  directly 
to  the  authors  or  to  me. 

Sharon  M.  Walter 
Rome  Laboratory/IRAA 
Griffiss  Air  Force  Base 
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1.  Introduction 

The  evaluation  of  natural  language  processing  (NLP)  systems  has  become  an  issue  of  in¬ 
creasing  concern  among  the  producers  and  consumers  of  NLP  products  as  the  number  of 
systems  and  applications  increases  [Neal91],  [Palmer89].  While  NLP  system  developers  look 
to  evaluation  as  a  means  of  measuring  capabilities  and  tracking  improvements  in  their  evolv¬ 
ing  systems,  consumers  are  concerned  with  comparative  evaluation  of  different  NLP  systems 
as  a  basis  for  selecting  NLP  systems  that  best  fit  the  communication  requirements  of  partic 
ular  applications.  NLP  producers  and  consumers,  as  weil  as  sponsors  of  NLP  research  and 
development,  also  see  evaluation  as  a  means  of  assessing  technical  progress  and  growth  in 
the  field. 

Currently,  some  of  the  most  common  applications  of  NLP  systems  are: 

•  Interactive  Human-Computer  Interfaces  to: 

-  Data  Base  Management  Systems 

-  Command  and  Control  Systems 

-  Decision-Aiding  Systems 

-  Engineering  Design  Systems 

—  Diagnostic  Systems 

•  Text  Understanding  for  Information  Extraction: 

—  Intelligence  Agencies 

—  DoD  Foreign  Technology  Divisions 

Figure  1  depicts  the  dilemma  of  application  system  developers  who  need  to  select 
an  appropriate  NLP  system  that  will  meet  their  needs.  In  the  process  of  developing  new 
application  systems,  analysis  of  requirements  produces  specifications  that  developers  need 
to  be  able  to  match  against  the  capabilities  of  existing  NLP  systems  in  order  to  identify 
those  NLP  systems  that  adequately  meet  the  system  requirements  for  NL  processing.  The 
NLP  Evaluation  Tool,  developed  a-r  part  of  the  Benchmark  Investigation/Identification  (I/I) 
Program,  supports  this  process  by  producing  detailed,  quantitative,  descriptive  evaluation 
profiles  of  NLP  systems  for  comparison  against  application  system  requirements.  The  clas¬ 
sification  scheme,  also  developed  as  part  of  the  Benchmark  I/I  Program,  provides  a  checklist 
of  language  processing  capabilities  for  use  in  determining  application  system  requirements. 
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REQUIREMENTS 


Figure  1:  Matching  Capabilities  of  NLP  Systems  Against  Requirements 

Figure  2  depicts  the  total  environment  of  the  problem.  The  wheel  in  the  figure  illus¬ 
trates  the  fact  that  NLP  systems  provide  users  with  the  means  or  support  for  performing 
application  domain  tasks.  The  user  is  represented  at  the  hub  of  the  wheel  and  the  circle 
around  the  hub  represents  the  different  types  of  NLP  systems,  including  those  for  text  and 
query  understanding,  text  generation,  dialogue  understanding,  and  speech  generation  and 
understanding.  The  main  spokes  of  the  wheel  are  the  types  of  linguistic  functionality  re¬ 
quired  in  NLP  systems  (phonological  being  limited  to  speech  systems).  Some  of  the  types  of 
phenomena  that  fall  within  the  given  functionality  area  are  listed  beneath  the  major  func¬ 
tionality  heading.  The  next  concentric  circle  of  the  wheel  is  interpreted  as  the  robustness 
factor,  which  represents  the  critical  functionality  for  dealing  with  unexpected  inputs,  both  er¬ 
rors  and  material  new  to  the  system,  as  well  as  systematic  acquisition  of  linguistic  knowledge 
for  extending  the  system.  Robustness  applies  to  all  the  areas  of  functionality.  The  outer¬ 
most  circle  of  the  wheel  represents  some  of  the  types  of  applications  that  an  NLP  system 
can  potentially  be  integrated  with,  including  command  and  control  systems,  decision-aiding 
systems,  planning  systems,  simulation  systems,  DBMSs,  etc. 

There  are  many  dimensions  along  which  NLP  systems  can  be  evaluated.  These  dimen¬ 
sions  include: 

•  linguistic  competence, 

•  end  user  issues  (e.g.,  habitability,  reliability,  likability), 
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Figure  2:  The  Context  of  the  NLP  Evaluation  Problem 


•  system  development  issues  (e.g.,  cost,  ease  of  development,  maintainability,  portabil¬ 
ity),  and 

•  intelligent  behavior  issues  (e.g.,  inference,  learning,  cooperative  dialogue). 

tiese  issues  are  mussed  in  more  detail  in  Section  2.  Of  the  many  dimensions  along  which 
LP  systems  can  be  evaluated,  this  project  focused  on  the  linguistic  competence  of  NLP 
stems,  including  lexical,  syntactic,  semantic,  and  discourse  capabilities. 

There  are  several  problems  in  evaluating  NLP  systems  with  regard  to  linguistic  com¬ 
mence.  First  is  the  need  for  clear  definitions  of  the  linguistic  phenomena  being  tested  and 
ie  classification  scheme  being  used.  For  example,  what  is  meant  by  the  claim  that  a  system 
mdles  “comparatives,”  “ellipsis”  or  “anaphoric  references”?  Since  there  are  various  types 
'  each  of  these  phenomena,  do  the  users  of  these  terms  mean  that  their  systems  handle  all 
rpes  of  the  particular  phenomenon,  or  just  a  subset? 

Second,  a  problem  that  has  occurred  with  most  evaluation  approaches  to  date  is  that 
ley  are  restricted  to  a  particular  application  (e.g.,  information  extraction,  database  re- 
ieval),  domain  (e.g.,  terrorism,  Navy  situation  reports,  company  employee  database  infor- 
lation),  and/or  NLP  system  type  (e.g.,  text  understanding  systems,  interactive  NL  front 
ids  to  application  systems).  For  example,  a  corpus-based  evaluation  approach  is  restricted 
)  the  specific  domain(s)  used  in  the  corpus  and  the  type  of  systems  for  which  the  corpus  was 
esigned.  Examples  of  corpora  that  have  been  developed  for  evaluation  of  database  query 
^sterns  include  the  BBN  corpus  [BBN88],  the  LADDER  corpus  [Hendrix76],  the  Malhotra 
irpus  [Malhotra75],  and  the  HP  corpus  [Flickinger87].  The  MUC-3  (Message  Understand- 
lg  Conference  #3)  [Sundheim91],  [Lehnert91]  evaluation  effort  and  the  more  recent  MUC-4 
VIessage  Understanding  Conference  #4)  were  also  restricted  to  a  particular  domain  and 
[LP  application.  They  used  a  domain  restricted  to  terrorism  activities  and  focused  on  the 
erformance  of  text  analysis  systems  for  information  extraction. 

These  application  domain  and  system-type  dependencies  can  cause  several  difficulties, 
irst,  for  a  domain-dependent  evaluation  method  such  as  the  MUCs,  the  task  of  porting 
[LP  systems  to  the  particular  domain  can  be  prohibitive.  This  may  limit  the  number 
f  participating  systems.  Furthermore,  deficiencies  of  the  participating  NLP  systems  being 
valuated  could  be  the  result  of  porting  difficulties  rather  than  problems  with  the  NLP  system 
self.  Second,  some  linguistic  phenomena  may  be  ignored  because  they  do  not  arise  in  the 
articular  application  domain  or  task  of  the  evaluation  exercise(s).  As  a  result,  linguistic 
apabilities  that  may  be  useful  for  other  applications  and  for  systems  of  the  future,  may  be 
jnored.  This  would  become  a  significant  problem  if  the  computational  linguistics  research 
Dmmunity  focused  too  strongly  on  certain  application  domains  and  tasks  to  the  exclusion 
f  others. 

Because  of  shortcomings  such  as  those  discussed  above,  the  goal  of  the  Benchmark 
ivestigation/Identification  Program  was  to  develop  a  methodology  and  procedure/tool  for 
valuating  NLP  systems  that  is,  insofar  as  possible,  independent  of  application,  domain,  and 
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r LP  system  type.  The  procedure/ tool  is  based  on  a  glossary  of  defined  terminology  and  a 
nguistic  phenomena  classification  scheme.  Read  et  al.  [Read88],  [Read90]  also  advocated 
bis  type  of  evaluation  methodology  in  discussions  of  their  work  on  the  Sourcebook.  Their 
ourcebook  provides  a  collection  of  exemplars  of  numerous  types  of  linguistic  phenomena,  but 
,  does  not  provide  a  procedure  or  method  for  evaluating  systems’  performance  on  processing 
tiese  phenomena. 

In  summary,  the  primary  objective  of  the  Benchmark  Investigation/Identification  Pro¬ 
ram  has  been  the  development  of  a  method  and  procedure/tool  for  evaluating  NLP  systems 
hat: 

1.  produces  profiles  of  NLP  systems  that  are: 

•  descriptive:  the  profiles  provide  descriptive  information  with  regard  to  the  types 
of  linguistic  phenomena  on  which  the  NLP  succeeded  or  failed,  not  just  one  or 
two  numerical  scores  (e.g.,  recall  and  precision)  that  provide  no  detailed  analysis. 

•  hierarchically  organized:  the  capabilities  of  NLP  systems  are  described  by  individ¬ 
ual  capability  as  well  as  by  class  of  capability,  at  the  various  levels  of  granularity 
provided  by  the  hierarchical  structure  of  the  profile. 

•  quantitative:  the  scores  that  are  assigned  by  the  evaluators  to  individual  test  items 
are  aggregated  by  class  and  weighted  averages  are  used  to  calculate  a  numerical 
performance  score  for  each  class  in  the  hierarchy. 

•  objective:  test  items  are  structured  in  a  detailed  manner  with  defined  criteria  so 
as  to  mitigate  the  subjectivity  of  the  evaluator. 

2.  is  usable  across  domains  (e.g.,  terrorism,  military  situation  reports,  company  employee 
database  information). 

3.  is  usable  across  applications  (e.g.,  text  processing  for  information  extraction,  message 
routing/dissemination,  human-computer  interface  for  interactive  dialogue) 

4.  is  applicable  across  the  different  types  of  NLP  systems  such  as  database  query  NL  front- 
end  systems,  text/message  processing  systems,  interactive  NL  dialogue  interfaces,  etc. 

5.  is  repeatable;  the  procedure/tool  produces  consistent  results,  independent  of  evaluator. 

6.  does  not  require  that  the  evaluator  be  a  trained  linguist. 

7.  is  unbiased  with  respect  to  linguistic  theories,  system-internal  processing  methods,  and 
knowledge  representation  techniques. 

The  NLP  Evaluation  Procedure  and  Tool  satisfy  the  objectives  listed  above  and,  in 
ict,  this  makes  the  NLP  Evaluation  Tool  unique.  In  particular,  the  features  that  are  unique 
o  the  Tool  are: 
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•  the  profiling  facility  and 

•  its  usability /a:  c!:  ability  across  domains,  applications,  and  system  types,  which  i.iea  ns 
that  the  Toe  .>es  not  require  NLP  systems  to  be  modified  or  ported  to  a  particular 
text  corpus  c?  domain. 

Since  the  NLP  Evaluation  Procedure/Tool  has  been  designed  to  produce  comprehensive 
le  oriptive  evaluation  profiles  for  NLP  systems,  such  a  profile  should  be  interpreted  in  terms 
>f  the  nature  of  the  NLP  system  type  and  the  application  task  for  which  the  system  is 
configured  or  installed  as  well  as  in  terms  of  the  requirements  of  the  application  for  which 
die  NLP  system  is  to  be  used.  One  would  not  necessarily  expect  a  particular  (type  of) 
SLP  system  to  excel  in  all  areas.  For  example,  a  text  processing  system  that  performs  well 
it  information  extraction  for  the  purpose  of  database  update  may  not  process  NL  queries 
Dr  commands.  On  the  other  hand,  a  database  query  NL  front-end  system  may  perform 
sxtremely  well  at  processing  NL  queries  and  commands,  but  may  not  process  declarative 
sentences. 

The  following  is  a  list  of  the  products  developed  as  part  of  the  Benchmark  Investiga¬ 
tion/Identification  Program: 

•  a  glossary  of  non-subjective  descriptive  terminology  for  describing  NLP  capabilities 
outside  the  context  of  their  application  to  target  software. 

•  a  bibliography  of  relevant  literature  used  in  this  effort. 

•  a  classification  scheme  for  NLP  capabilities  and  issues  that  provides  the  hierarchical 
organization  for  NLP  system  profiles. 

•  the  Tool/ Procedure  which  guides  the  evaluator  through  the  evaluation  process.  This 
Tool  provides  assistance  to  the  evaluator  with  the  development  of  test  sentences  and 
provides  for  the  recording  of  results/scores. 

•  the  profiling  mechanism  for  producing  descriptive  profiles  of  NLP  systems.  The  pro¬ 
filing  facility  can  describe  a  system’s  ability  to  process  linguistic,  phenomena  in  terms 
of  fairly  coarse-grained,  broadly  defined  classes  of  phenomena  as  well  as  in  terms  of 
detailed,  fine-grained,  narrowly  defined  classes  of  phenomena. 

•  periodic  assessment  of  the  NLP  Evaluation  Tool  at  the  end  of  each  of  the  three  six- 
month  development  phases  of  the  program.  This  assessment  consisted  of  having  inter¬ 
face  technologists,  who  were  not  linguists,  apply  the  NLP  Evaluation  Procedure/Tool 
to  several  well-developed  NLP  systems.  The  interface  technologists  had  no  involvement 
in  the  development  of  the  Tool,  but  received  training  in  its  application.  The  results  of 
the  assessment  by  the  interface  technologists  provided  feedback  to  the  developers  dur¬ 
ing  its  incremental  development.  Among  the  systems  used  for  the  assessment  activity 
are  three  commercially  available  NL  database  query  systems  and  three  of  the  MUC-3 
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participant  systems.  The  final  assessment  of  the  Tool  provided  information  regarding 
the  reliability  of  the  Tool. 

This  report  discusses  the  products  of  the  Benchmark  Investigation/Identification  effort, 
articularly  the  NLP  Evaluation  Tool,  and  the  final  assessment  experiment  involving  the 
Evaluation  Tool.  Section  2  discusses  background  and  scope.  Section  3  briefly  discusses 
he  bibliography  developed  as  part  of  the  effort  and  Section  4  briefly  covers  the  glossary  of 
escriptive  terminology.  Section  5  di  usses  the  classification  scheme.  Section  6  discusses 
he  design,  content,  organization,  and  use  of  the  Evaluation  Tool.  Section  7  discusses  the 
irofiling  facility  and  the  profiles  produced  by  the  facility.  Section  8  reports  on  the  final 
-ssessment  of  the  Evaluation  Tool.  Section  9  provides  conclusions  and  recommendations. 

,  tion  10  includes  the  references  used  in  this  report.  Appendix  A  contains  the  complete 
‘sLP  Evaluation  Tool.  Appendix  B  includes  a  profile  of  an  NLP  system  produced  by  the 
-iLP  Evaluation  Tool.  Appendix  C  includes  a  summary  of  the  Assessment  Questionnaire 
esponses  from  the  interface  technologists. 
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2.  Background 

There  are  many  different  areas  and  issues  for  which  NLP  systems  need  to  be  evaluated.  Table 
1  categorizes  and  lists  many  of  these  issues.  The  problem  of  evaluation  of  NLP  systems  is  dif¬ 
ficult  and  many  approaches  to  these  different  issues  have  been  discussed  [Bates90],  [BBN88], 
[Biermann83],  [Flickinger87],  [Guida86],  [Hayes-Roth89],  [Hendrix76],  [Hershman79],  [Hix9l], 
[Kohoutek84],  [Lazzara90],  [Malhotra75],  [Mitta91],  [Ogden88],  [Palmer89],  [Read88],  [Read90], 
[Sundheim91],  [Tennant79j,  [Weischedel86].  The  NLP  Evaluation  Tool,  developed  as  part 
of  the  Benchmark  I/I  Program,  focuses  on  the  linguistic  issues  listed  in  the  first  column  of 
the  table.  The  following  paragraphs  briefly  review  some  of  the  related  evaluation  efforts  and 
approaches. 


Table  1:  Categories  of  Evaluation  Issues 


Linguistic 

Issues 

Intelligent  Behavior 
k  Reasoning  Issues 

End  User 
Issues 

System  Development 
Issues 

lexicon 

syntax 

semantics 

discourse 

pragmatics 

inference 

learning 

cooperative  dialogue 
speaker/ hearer  modeling 
real  world  knowledge 

habitability 

reliability 

likability 

efficiency 

extensibility 

quality  of  tools 
cost 

ease  of  development 
maintainability 
portability 
integrability 

Several  studies  have  focused  on  the  issue  of  habitability.  Hershman,  Kelly,  and  Miller 
[Hershman79]  conducted  laboratory  evaluations  in  which  these  researchers  studied  ten  Navy 
officers  using  LADDER,  a  natural  language  query  system  designed  to  provide  easy  access  to  a 
naval  database.  The  study  simulated  the  actual  operational  environment  in  which  LADDER 
would  be  used  and  the  subjects  were  trained  to  the  database  and  LADDER  interface.  The 
results  of  the  study  indicated  that  the  extensive  training  given  to  the  subjects  was  adequate 
for  training  the  functional  and  conceptual  coverage  of  the  system  but  not  for  training  the 
syntactic  and  lexical  coverage. 

Focusing  on  habitability  and  efficiency,  Biermann,  Ballard,  and  Sigmon  [Biermann83] 
designed  an  experiment  that  was  concerned  with  the  usefulness  of  English  as  a  programming 
language.  Their  experiment  used  a  natural  language  programming  system,  called  NLC,  that 
allows  a  user  to  display  and  manipulate  tables  and  matrices  while  at  a  display  terminal. 
All  user  inputs  were  expressed  in  English.  The  results  of  the  study  indicated  that,  with 
a  relatively  short  training  period  on  the  NLC  system,  subjects  were  able  to  type  system- 
acceptable  syntax  with  a  high  enough  success  rate  to  obtain  correct  answers  in  a  reasonable 
amount  of  time. 

The  Performance  Evaluation  Plan  (PEP)  [Lazzara90]  addresses  the  evaluation  of  NLP 
tools  or  shells  for  the  development  of  specific  NLP  applications  from  a  user-oriented  perspec¬ 
tive,  where  three  classes  of  users  are  identified:  systems  developers,  end  users,  and  systems 
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maintainers.  As  a  result,  the  PEP  provides  a  methodology  for  evaluating  issues  such  as 
integrity,  maintainability,  extendability,  portability,  user  productivity  and  likability. 

Hayes-Roth  [Hayes- Roth89]  and  Mitta  [Mitta91]  are  concerned  with  evaluation  of 
knowledge  systems  and  expert  systems.  Hayes-Roth  [Hayes-Roth89]  is  primarily  concerned 
with  extrinsic  issues  such  as  advice  quality,  reasoning  correctness,  robustness,  solution  effi¬ 
ciency  and  intrinsic  issues  such  as  elegance  of  knowledge  base  design,  modularity,  and  archi¬ 
tecture.  Mitta  [Mitta91]  discusses  a  methodology  for  evaluating  an  expert  system’s  usability, 
based  on  the  following  six  variables  or  measures:  user  confidence  that  the  solution  is  correct, 
user  perception  of  difficulty,  correctness  of  solution,  number  of  responses  required  of  users, 
inability  of  expert  system  to  provide  a  solution,  and  the  rate  of  help  requests.  Although 
focusing  on  knowledge  systems  or  expert  systems,  these  discussions  and  methodologies  have 
applicability  to  NLP  systems  also,  since  NLP  systems  are  special  types  of  knowledge  systems. 

Several  approaches  and  studies  focus  on  linguistic  and  NL  understanding  capabilities. 
Guida  and  Mauri  [Guida86]  have  developed  a  formal  and  detailed  method  for  evaluating 
NLP  systems.  They  treat  an  NLP  system  as  a  function  from  a  set  of  input  expressions  to 
one  or  more  sets  of  outputs.  Their  method  requires  a  measure  of  error,  defined  to  compare 
the  closeness  of  the  output  to  the  correct  output,  and  a  measure  of  the  importance  of  each 
input.  Their  method  computes  the  sum  of  the  errors  weighted  by  the  importance  of  the 
input,  as  the  evaluation  of  an  NLP  system. 

Several  approaches  that  focus  on  linguistic  capabilities  have  entailed  the  development 
of  test  corpora  for  evaluating  NL  database  query  interfaces  [BBN88],  [Hendrix76],  [Malho- 
tra75],  and  [Flickinger87].  Flickinger,  Nerbonne,  Sag,  and  Wasow  [Flickinger87]  developed 
a  test  suite  of  English  sentences,  annotated  by  construction  type,  that  covers  a  wide  vari¬ 
ety  of  syntactic  and  semantic  phenomenon.  In  order  to  avoid  theory  dependence,  the  level 
of  granularity  in  the  test  suite  was  built  on  intuition  rather  than  theoretical  significance. 
Anomalous  strings  are  included  as  well  as  well-formed  sentences. 

As  part  of  the  Artificial  Intelligence  Measurement  System  (AIMS)  project  [Read88], 
evaluation  criteria  and  methods  for  describing  linguistic  coverage  were  developed  for  NLP 
systems.  As  a  result,  a  Sourcebook  [Read90]  was  developed  consisting  of  a  database  of 
‘exemplars’  of  representative  problems  in  NL  processing.  Each  exemplar  includes  a  piece 
of  illustrative  text,  a  description  of  the  linguistic/conceptual  issue  at  stake,  discussion  of 
the  problems  in  understanding  the  text,  and  references  to  more  extensive  discussion  in  the 
literature. 

The  Naval  Ocean  Systems  Center  (NOSC)  completed  the  third  evaluation  of  English 
text  processing  systems  in  May,  1991,  with  the  Third  Message  Understanding  Conference 
(MUC-3)  [Sundheim91].  The  Fourth  Message  Understanding  Conference  (MUC-4)  took 
place  16-18  June  1992  (no  report  available  yet).  These  evaluations  have  focused  on  the 
performance  of  text  analysis  systems  on  an  information  extraction  task.  The  training  data 
set  consisted  of  1300  texts  with  an  overall  size  of  over  2.5  megabytes.  The  task  was  to  extract 
information  on  terrorist  incidents  from  relevant  text  among  100  previously  unseen  texts. 
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Finally,  an  important  issue  is  the  reliability  of  evaluation  methods.  That  is,  different 
evaluators  must  produce  consistent  results  when  applying  the  same  evaluation  method  to 
the  same  target  system.  Although  not  concerned  with  NL  processing  directly,  the  approach 
of  Hix  and  Schulman  [Hix91]  for  testing  the  reliability  of  their  methodology  for  evaluating 
human-computer  interface  development  tools  is  relevant.  To  empirically  test  their  methodol¬ 
ogy,  Hix  and  Schulman  had  six  evaluators  each  apply  the  method  to  two  (out  of  a  total  three) 
application  tools,  so  that  each  tool  was  evaluated  by  four  different  participants.  To  produce 
statistical  tests  of  reliability,  the  researchers  computed  the  probability  that  responses  from 
the  four  evaluators  for  each  tool  would  match  by  chance.  The  observed  proportion  of  matches 
for  each  category  of  items  was  compared  with  the  chance  probability  using  a  binomial  test. 

The  NLP  Evaluation  Procedure  focuses  on  the  linguistic  capabilities  of  NLP  systems. 
Important  components  of  the  NLP  Evaluation  Tool  are  the  hierarchically  structured  clas¬ 
sification  scheme  for  linguistic  phenomena,  the  emphasis  on  descriptions  of  the  linguistic 
phenomena  covered  in  the  Tool,  and  the  examples  illustrating  linguistic  phenomena.  In 
these  aspects,  it  has  some  similarity  to  the  Sourcebook  approach  of  Read  et  al  [Read88],  but 
the  NLP  Evaluation  Tool  provides  broader  and  more  detailed  coverage  than  the  Sourcebook. 
Furthermore,  the  Benchmark  I/I  Program  product  includes  a  procedure  or  tool  for  testing 
whether  NLP  systems  are  capable  of  handling  the  described  linguistic  phenomena,  which 
the  Sourcebook  does  not.  The  NLP  Evaluation  Tool  includes  patterns,  instructions,  and 
illustrative  examples  for  composing  NL  text  for  testing  purposes  and  a  profile  generator  that 
displays  the  evaluation  results  in  the  form  of  descriptive  profiles  of  NLP  systems  organized 
according  to  the  hierarchically  structured  classification  scheme  for  linguistic  phenomena.  We 
also  conducted  a  reliability  experiment  for  the  NLP  Evaluation  Tool  that  has  some  similarity 
to  that  of  Hix  and  Schulman  [Hix91|.  In  contrast  to  the  approaches  discussed  in  this  section, 
with  the  exception  of  the  Sourcebook  [Read90],  the  NLP  Evaluation  Tool  is  designed  to  be 
applicable  to  different  types  of  NLP  systems  and  to  be  applicable  across  application  domains. 

The  following  sections  discuss  the  Benchmark  I/I  Program  products:  the  bibliogra¬ 
phy,  the  glossary  of  descriptive  terminology,  the  classification  scheme,  the  NLP  Evaluation 
Procedure/Tool,  and  the  profiling  facility. 
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3.  The  Bibliography 

As  part  of  the  Benchmark  I/I  Program,  we  developed  a  bibliography  of  literature  that  served 
as  the  basis  for  the  terminology  definitions  and  the  development  of  test  items  for  the  NLP 
Evaluation  Tool.  The  bibliography  was  developed  from  literature  search  and  review  and  has 
approximately  600  entries.  The  bibliography  is  included  in  Volume  II  of  this  report. 

We  performed  a  search  for  relevant  literature  from  several  databases.  The  research  team 
used  services  such  as  the  DIALOG  Information  Retrieval  Service,  from  Dialog  Information 
Services,  and  the  ORBIT  Search  Service,  from  the  Pergamon  Group  of  Companies  as  well  as 
databases  that  are  not  part  of  these  services.  The  literature  databases  that  were  used  include 
the  MLA  (Modern  Language  Association)  database,  the  BISON  (Buffalo  Information  System 
OnliNe)  database,  LINGUISTICS  &  LANGUAGE  BEHAVIOR  ABSTRACTS,  INSPEC, 
NOTIS,  COMPENDIX  PLUS,  The  COMPUTER  DATABASE,  and  PsycINFO. 

Other  literature  sources  that  were  used  include  recent  publications  of  the  Association 
for  Computational  Linguistics  (ACL),  the  American  Association  for  Artificial  Intelligence 
(AAAI),  and  the  Linguistics  Society  of  America  (LSA). 

4.  The  Glossary  of  Descriptive  Terminology 

Identification  and  definition  of  the  evaluation  issues  and  criteria  is  important  in  any  evalua¬ 
tion  effort.  The  Benchmark  I/I  Program  focused  on  the  ability  of  NLP  systems  to  process  the 
various  constructs  and  phenomena  of  natural  language.  As  part  of  this  project,  we  developed 
a  glossary  of  descriptive  terminology  with  which  to  describe  the  linguistic  phenomena  and 
features  for  which  NLP  systems  are  tested  in  the  NLP  Evaluation  Tool.  This  terminology 
was  developed  from  the  literature  on  linguistics  and  computational  linguistics.  Definitions 
are  based  on,  or  selected  from,  well-respected  literature  sources.  This  terminology  is  used 
throughout  the  Tool  to  identify  what  is  being  evaluated  in  each  Tool  item  and  it  is  used  in 
the  classification  scheme  that  provides  the  framework  for  the  NLP  system  profiles  produced 
by  the  Tool.  The  Glossary  is  included  in  Volume  II  of  this  final  report. 

5.  The  Classification  Scheme 

As  part  of  the  Benchmark  I/I  Program,  we  developed  a  hierarchical  classification  scheme 
for  linguistic  phenomena  that  also  serves  as  the  organizing  framework  for  displaying  NLP 
system  profiles.  This  classification  scheme  has  been  designed  to  accommodate  a  spectrum 
of  concepts  from  broadly  defined  categories  of  phenomena  to  specific  types  of  phenomena. 

One  of  the  problems  with  classification  of  linguistic  phenomena  is  that  the  phenomena 
frequently  cannot  be  categorized  into  unique  categories,  but  should  be  classified  in  multiple 
categories.  For  example,  comparatives  commonly  include  the  use  of  ellipsis,  e.g.,  “Is  John 
as  old  as  Dave?”  Should  this  type  of  linguistic  phenomenon  be  classified  as  ellipsis,  com¬ 
paratives,  both,  or  in  an  elliptical-comparatives  class  that  may  be  a  subclass  of  both  ellipsis 
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and  comparatives? 

Another  problem  is  in  determining  the  appropriate  categories  of  test  phenomena  for 
a  black  box  evaluation  method.  For  example,  for  the  four  types  of  processing  (e.g.,  lexi¬ 
cal,  syntactic,  semantic,  and  discourse)  typically  identified  in  NLP  systems,  should  each  of 
the  four  be  separately  identified  for  each  linguistic  phenomenon?  For  example,  handling 
pronominal  anaphoric  references  would  typically  entail  lexical  capabilities  such  as  recogniz¬ 
ing/determining  the  lexical  features  of  the  pronoun,  recognition  of  the  pronoun  as  a  syntactic 
component  of  the  clause  or  phrase  being  processed,  use  of  semantic  knowledge  about  other 
entities  in  the  discourse,  and  the  use  of  discourse  knowledge  in  being  able  to  select  from 
among  the  candidate  referents  for  the  pronoun.  Should  each  of  these  component  types  of 
capabilities  be  identified  in  the  classification  scheme  (e.g.,  classes  called  pronoun  anaphora  - 
lexical ,  pronoun  anaphora  -  syntactic ,  pronoun  anaphora  -  semantic ,  and  pronoun  anaphora 
-  discourse )? 

This  type  of  categorization  scheme,  however,  is  not  appropriate  if  a  black  box  approach 
is  being  used  on  NLP  systems,  treated  as  whole  systems,  and  access  to  the  components  is 
not  part  of  the  evaluation.  For  example,  if  a  system  fails  on  a  particular  test  item,  such  as 
pronoun  anaphoric  references,  it  is  very  difficult  to  determine  which  of  the  four  component 
capabilities  really  caused  the  processing  failure. 

Since  the  goals  of  the  Benchmark  I/I  Program  were  focused  on  evaluation  of  whole 
NLP  systems  rather  than  individual  components,  our  approach  was  to  develop  test  items 
that  address  the  detailed  individual  component  capabilities  to  the  best  of  our  ability,  while 
treating  the  systems  as  whole  systems  and  not  expecting  to  examine  the  inputs  and  outputs 
of  any  system’s  components.  We  also  opted  for  a  tree  structured  classification  scheme  that 
includes  verbal  pointers  to  other  sections  of  the  classification  scheme  for  classes  that  could 
be  classified  under  multiple  parents.  Figure  3  shows  the  classification  hierarchy  for  a  portion 
of  the  section  on  Reference  issues.  The  entire  classification  scheme  developed  under  the 
Benchmark  I/I  Program  is  displayed  in  the  NLP  system  profile  in  Appendix  B. 
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X.  REFERENCE 

1.  Specific  Reference 

1.1  Anaphoric  Reference 

1.1.1  Pronominal  Anaphora 

1.1. 1.1  Anaphoric  References  With  NP  Antecedents 

1.1. 1.1.1  Anaphoric  Reference  Functions  as  Subject 

1.1. 1.1.2  Anaphoric  Reference  Functions  as  Object 

1.1. 1.1. 3  Anaphoric  Reference  Functions  as  Possessive 

1.1. 1.1.4  Ambiguous  Anaphoric  References 

1.1. 1.2  Anaphoric  References  with  Verb  Phrase  or  Sentence  Antecedents 

1.1.2  Nominal  Anaphora 

1.1. 2.1  Anaphoric  References  with  NP  Antecedents 

1.1. 2. 1.1  Direct 

1.1. 2. 1.2  Indirect 

1.1. 2.2  Anaphoric  References  With  Clause  or  Sentence  Antecedents 

1.1.3  Anaphora  with  SO  and  AS 

1.1. 3.1  Anaphora  with  SO 

1. 1.3.2  Anaphora  with  AS 

1.1.4  Intra-  and  Inter-Sentential  Anaphora 

1. 1.4.1  Intra-sentential  anaphora 

1.1. 4.2  Inter-sentential  anaphora 

1.2  Cataphoric  Reference 

1.2.1  Cataphoric  References  With  NP  Antecedents 

1.2.2  Cataphoric  References  With  Verb  Phrase  or  Sentence  Antecedents 

1.3  General  Knowledge  or  the  Larger  Situation 

2.  Generic  Reference 


Figure  3:  The  Classification  Scheme  for  Section  X  Reference 
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6.  The  NLP  Evaluation  Procedure/Tool 

The  organization  of  the  NLP  Evaluation  Tool  is  discussed  in  Section  6.1,  the  Tool  design 
is  discussed  in  Section  6.2,  and  the  computer-based  Tool  application  facility  is  discussed  in 
Section  6.3.  The  Tool  includes  352  items,  each  covering  a  different  linguistic  phenomenon. 
Tool  items  are  classified  according  to  the  classification  scheme  discussed  in  Section  5. 

6.1  Tool  Organization 

The  top  level  of  the  classification  scheme  consists  of  the  categories  listed  below.  These 
categories  are  also  the  topics  of  the  twelve  major  sections  of  the  Tool,  as  shown  in  Table  2. 


Table  2:  The  Twelve  Main  Tool  Sections 


SECTION 

NUMBER 

SECTION 

TITLE 

I. 

Basic  Sentences 

II. 

Interrogative  Sentences 

III. 

Noun  Phrases 

IV. 

Adverbials 

V. 

Verbs  and  Verb  Phrases 

VI. 

Quantifiers 

VII. 

Comparatives 

VIII. 

Connectives 

IX. 

Embedded  Sentences 

X. 

Reference. 

XL 

Ellipsis 

XII. 

Semantics  of  Events 

The  Tool  starts  with  basic  sentences  since  the  input  to  most  NLP  systems  must  consist 
of  whole  sentences,  except  for  cases  of  ellipsis  which  typically  follow  complete  sentences.  The 
first  chapter  covers  the  basic  sentence  types  (i.e.,  declaratives,  interrogatives,  and  impera¬ 
tives),  basic  determiners  (i.e.,  articles),  basic  noun  phrases,  and  basic  verb  phrases.  Starting 
in  Tool  Section  I,  the  evaluator  determines  the  capabilities  of  the  system  that  can  be  used 
in  subsequent  test  inputs,  so  as  to  focus  each  test  on  the  particular  targeted  phenomenon, 
using  the  phenomenon  in  the  context  of  other  linguistic  phenomena  upon  which  the  system 
has  already  succeeded. 

6.2  Tool  Design 

Since  a  goal  of  our  NLP  Evaluation  Tool  design  effort  is  to  achieve  domain  or  application 
independence ,  insofar  as  possible,  the  Tool  is  designed  so  that  it  does  not  rely  on  a  particular 
corpus  of  natural  language  text  or  sentences.  Instead,  the  test  sentences  or  paragraphs  to  be 
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processed  by  an  NLP  system  are  composed  by  the  evaluator  either  during,  or  prior  to,  the 
administration  of  the  evaluation  procedure/test  to  the  NLP  system.  The  NLP  Evaluation 
Tool  is  designed  to  assist  the  evaluator  with  the  creation,  modification,  or  tailoring  of  test 
sentences. 

In  order  to  make  the  Tool  sensitive  to  individual  linguistic  capabilities,  it  is  designed 
so  that,  for  untested  individual  NLP  capabilities,  each  item  tests  just  one  NLP  capability  at 
a  time,  to  the  extent  possible,  and  combinations  are  tested  after  the  individual  capabilities 
are  tested.  The  Tool  is  designed  to  progress  from  very  elementary  sentence  types  containing 
simple  constituents  to  more  complex  sentence  (or  paragraph)  types.  The  idea  is  that  each 
time  a  test  sentence  (or  paragraph)  is  presented  to  the  NLP  system  being  evaluated,  the 
sentence  (or  paragraph)  should  contain  only  one  new  (untested)  linguistic  capability  or  one 
new  untested  combination  of  tested  capabilities.  The  other  capabilities  required  for  pro¬ 
cessing  the  input  should  already  have  been  tested  and  the  NLP  system  should  already  have 
succeeded  on  these  other  issues.  In  administering  the  NLP  Evaluation  Tool,  the  evaluator 
must  avoid  combining  tests  for  several  capabilities  in  the  same  test  sentences,  since  the  Tool 
would  then  be  insensitive  to  the  individual  capabilities.  For  example,  a  test  of  ellipsis  only 
in  the  context  of  question-answering  dialogue  would  not  be  usable  with  a  system  that  is  not 
designed  to  handle  questions  (e.g.,  a  text  understanding  system  designed  for  an  information 
extraction  task,  which  typically  processes  declarative  sentences,  but  not  interrogatives  or 
imperatives). 

Each  Tool  item  includes: 

•  An  explanation  and  definition  of  the  linguistic  phenomenon  or  capability  being  tested, 
along  with  any  special  instructions  for  testing. 

•  Patterns/descriptions  that  define  the  structure  and  features  of  the  test  sentences  to  be 
composed  and  input  to  the  NLP  system  under  evaluation. 

•  Example  sentences  to  aid  the  evaluator  in  composing  test  sentences. 

•  A  statement  of  the  evaluation  criteria  for  the  individual  test  item. 

•  A  place  for  the  evaluator’s  test  sentences. 

•  A  place  for  the  evaluator’s  score. 

Figure  4  shows  a  Tool  item  from  Section  2.2  that  addresses  the  issue  of  adjective 
nominals.  The  item  begins  with  an  explanation  or  definition  of  the  linguistic  phenomenon 
being  tested  along  with  instructions  for  composing  appropriate  test  inputs.  Four  suggested 
patterns  are  then  given  to  help  guide  the  evaluator  in  creating  test  input  sentences.  Below 
each  pattern  is  an  example.  The  examples  in  the  Tool  are  from  two  application  domains: 
an  employee  database  information  retrieval  domain  and  the  terrorist  domain  of  the  MUC- 
3  exercise.  The  evaluator  should  pick  a  sentence  pattern  from  among  thos**  listed  in  the 
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particular  test  item,  or  from  among  those  in  previous  sections  upon  which  the  NLP  system 
has  succeeded.  The  evaluator  is  expected  to  make  the  substitutions  described  in  the  Tool  item 
instructions,  namely,  to  replace  the  “[NP]”  with  “([Det])  [Adj.Nom.]”.  Then  the  evaluator  is 
to  replace  the  non-terminals  in  the  pattern  with  words  that  are  within  the  lexicon  of  the  NLP 
system  being  tested  and  appropriate  to  the  domain  of  the  system.  So,  for  example,  if  the 
domain  of  a  system  was  mission  planning  for  the  Air  Force,  an  appropriate  test  input  might 
be  “List  the  most  senior  in  the  Iraqi  military,”  which  matches  the  first  sentence  pattern 
with  “the  most  senior”  as  the  adjective  nominal.  The  test  criteria  is  stated  so  as  to  focus 
the  evaluator’s  attention  on  the  critical  phenomenon  being  tested  and  to  provide  a  standard 
(objective)  scoring  metric.  Then  below  the  criteria  is  a  place  for  the  evaluator’s  test  input 
and  score. 

For  each  test  item  in  the  NLP  Evaluation  Tool,  the  evaluator  submits  an  NL  input  to 
the  NLP  system  being  evaluated  and  determines  whether  or  not  the  response  indicates  that 
the  system  understood  and  processed  the  input  correctly.  The  evaluator  has  five  choices  of 
scores  to  award  for  each  test  input: 

•  Success:  The  system  successfully  met  the  evaluation  criteria  stated  for  the  particular 
test  item. 

•  Correct:  The  system  did  not  successfully  meet  the  evaluation  criteria,  but  produced 
acceptable/correct  output. 

•  Partially  Correct:  The  system  did  not  successfully  meet  the  evaluation  criteria,  and 
only  produced  partially  acceptable/correct  output. 

•  Failure:  The  system  did  not  successfully  meet  the  evaluation  criteria  and  produced  no 
correct  output. 

•  No  output:  The  system  produced  no  output. 

One  of  the  difficulties  in  applying  this  NLP  Evaluation  Tool  is  that  the  decision  as  to 
what  score  should  be  awarded  to  the  system  for  each  test  item  can  be  somewhat  subjective. 
That  is,  there  is  no  predefined  answer  key  that  must  be  matched  for  each  test  input.  Instead, 
the  evaluator  is  expected  to  create  good,  discriminating  test  inputs  and  define  the  correct 
outputs  based  on  the  criteria,  just  as  a  teacher  would  for  a  class  of  students.  Therefore,  the 
evaluator  must  be  trained  in  the  application  domain  and  task  of  the  NLP  system  and  must 
know: 

•  the  vocabulary  of  the  NLP  system  so  as  to  stay  within  its  bounds,  unless  the  vocabulary 
itself  is  the  focus  of  a  test  item; 

•  the  tasks  that  the  NLP  system  (with  its  back-end  system,  if  appropriate)  is  capable  of 
performing  (e.g.,  retrieving  data  from  a  database); 
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2.2  Nominals 

Nominate  are  non-noun  parts  of  speech  used  as  the  head  of  a  noun  phrase. 

2.2.1  Adjective  Nominal 

An  adjective  nominal  consists  of  an  optional  determiner  followed  by  an  adjective  used 
as  the  head  of  the  noun  phrase.  Examples:  “the  elderly”,  “the  poor”,  “the  youngest”, 
“the  smartest”.  You  may  use  any  of  the  successful  sentence  patterns  from  Sections  I  or 
II  to  test  an  adjective  nominal.  Replace  the  “[NP]”  in  the  chosen  sentence  pattern  with 
“([Det])  [Adj.Nom.]”.  We  use  the  notation  “[Det]”  to  denote  the  determinative  element  and 
“[Adj.Nom.]”  to  denote  the  adjective  nominal.  Below  are  some  suggested  patterns. 

Pattern:  [Verb]  ([Det])  [Adj.Nom.]  ([PP]) 

Example:  List  the  oldest  in  the  New  York  branch. 

Pattern:  [WH-Word]  [BE-Verb]  ([Det])  [Adj.Nom.]  ([PP]) 

Example:  Who  is  the  oldest  in  Chicago? 

Pattern:  [NP]  [Verb]  ([Det])  [Adj.Nom.]  ([PP]) 

Example:  Terrorists  killed  the  elderly  in  Usulutan. 

Pattern:  ([Det])  [Adj.Nom.]  ([PP])  [Verb]  [NP]. 

Example:  The  poor  in  Usulutan  attacked  the  Government 

House  . 

Criteria:  Demonstrated  understanding  by  interpreting  the  adjective  nominal  as  expressing 
an  entity  or  object. 

Input:  Score: 


Figure  4:  The  Adjective  Nominal  Tool  Item 
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•  the  concepts,  objects,  and/or  attributes  in  the  domain  that  can  be  the  focus  of  test 
inputs  (e.g.,  one  can  query  an  employee  database  about  certain  attributes  of  the  persons 
in  the  database,  such  as  their  branch,  pay,  sales,  longevity  with  the  company). 


6.3  Tool  Application  Facility 

In  order  to  facilitate  the  use  of  the  Tool,  we  developed  an  Emacs-based  dual-window  split¬ 
screen  facility  for  an  evaluator  to  use  when  applying  the  Evaluation  Tool  to  an  NLP  system. 
The  facility  provides: 

•  inter-window  text  manipulations  and 

•  storage  of  the  completed  (or  partially  completed)  Evaluation  Tool  sections  to  file. 

The  dual-window  interface  is  implemented  on  top  of  Emacs  and  presents  the  user  with 
two  tiled  Emacs  buffer  windows  above  one  another  as  shown  in  Figure  5.  The  Evaluation 
Tool  is  visible  and  accessible  in  the  upper  window  and  the  particular  NLP  system  being 
evaluated  is  accessible  in  the  lower  window.  Figure  5  shows  the  Tool  application  facility 
being  used  on  a  SUN  SPARCstation  with  a  MUC-3  text  processing  system. 

The  Tool  application  facility  supports  the  evaluator  in  applying  the  Tool  to  an  NLP 
system  and  in  recording  test  inputs,  assigned  scores,  system  outputs,  and  comments  within 
the  on-line  version  of  the  Tool  for  the  NLP  system.  The  Emacs-based  Tool  application 
facility  enables  an  evaluator  to  edit  text  in  either  window  and  copy  blocks  of  text  between 
the  two  windows.  This  allows  the  evaluator  to  edit  text  in  the  Tool  window  and  to  execute 
the  NLP  system  using  the  text  copied  from  the  Tool  window  as  input.  The  evaluator  can 
also  mark  and  copy  text  back  from  the  NLP  system  window  to  the  Tool  window  to  record 
selected  system  outputs.  The  evaluator  records  his/her  score  and  comments  for  each  Tool 
item  in  the  Tool  window.  This  facility  has  proven  to  be  a  great  convenience  to  the  evaluator 
in  evaluation  experiments  with  the  Tool. 

The  Evaluation  Tool  is  stored  on-line  with  each  major  section  in  a  different  file.  Each 
file  is  named  using  the  Roman  numeral  of  the  section  number.  When  an  evaluator  uses  the 
application  facility,  he/she  would  use  just  one  Tool  file  (major  section)  at  a  time.  When 
an  evaluator  wishes  to  quit  an  evaluation  session,  he/she  can  logout  of  the  Tool  application 
facility  and  the  Evalution  Tool  file  is  saved  with  the  evaluator’s  additions,  comments,  scores, 
etc.,  and  is  named  with  the  Roman  numeral  of  the  particular  section  and  the  evaluator’s 
initials  as  the  file  name  extension. 

The  Emacs-based  Tool  application  facility  for  a  Sun  SPARCstation  is  available  from 
the  authors. 
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The  existential  quantifier  "bom"  expresses  a  part  of  an  object  or  an 
unspecified  nwbec  of  objects. 

Eg,  List  |«om  employees  in  the  New  York  office]. 

Eg,  Mhat  p coduct a  do  (bom  of  the  salespersons]  sell7 
Eg,  (Bom  of  the  Cabinet  Members]  were  killed  by  a  carbomb. 

Eg,  Terrorists  killed  [som  peasants  in  cullapa]. 

Eg,  (Prel  lm  inary  Sentence:  rive  fa  ms  were  attacked  by  URMG 
guerrillas.)  [Som  fames]  were  destroyed  in  the  attack. 

Eq.  (Som  dynamite]  was  stolen  by  URMG  guerrillas  on  February  5. 

Criteria:  Demonstrated  understanding  that  the  existential  quantifier  "s 
expresses  a  part  of  an  object  or  an  unspecified  nvseber  of  objects. 


2.2.2  SEVERAL 

The  existential  quantifier  "several**  expresses  more  than  two  but  fewer  than 
ny. 


-  Enters :  bonciuiM  ik.lo(|.VI.Qif  l*»  i  Jim  S  4  ■  4  4. on  H.» i 1  ( T  t  Miluov  Fill) 41 


£6 

THU| 

! 

gEi 

1 

1 

>  (nit  "  Terrorists  killed  som  peasants  in  Cullapa.") 

NIL 

>  “*  Applying  perpetrator  heuristics 

C- DEATH -TEMP LATE  :  Individuals:  terrorist  <C-VERB_TERR0RI2E1-ER)  authorities  NIL  other\ 

s  NIL 

Organizations:  terrorist  NIL  authorities  NIL  others  NIL 

Conf idsnce(s) : 

0. 

MESSAGE  ZD 

DEV-HUC3 -0000 

1. 

TEMPLATE  ID 

1 

2. 

DATE  OF  INCIDENT 

-  31  oct  as 

3. 

TYPE  OF  INCIDENT 

MURDER 

4. 

CATEGORY  OF  INCIDENT 

TERRORIST  ACT 

5. 

PERPETRATOR:  ID  OF  INDIV(8) 

"Terrorists" 

4. 

PERPETRATOR:  ID  OF  ORC(S) 

7. 

PERPETRATOR:  CONFIDENCE 

8. 

PHYSICAL  TARGET:  ID (8) 

9. 

PHYSICAL  TARGET:  TOTAL  MUM 

10. 

PHYSICAL  TARGET:  TYPE(S) 

it. 

HUMAN  TARGET:  ID(8) 

"peasants" 

12. 

HUMAN  TARGET:  TOTAL  HUM 

PLURAL 

13. 

HUMAN  TARGET:  TYPE (8) 

CIVILIAN:  "peasants" 

14. 

TARGET:  FOREIGN  HATIOH(S) 

- 

15. 

INSTRUMENT:  TYPE (8) 

- 

16. 

LOCATION  OF  INCIDENT 

GUATEMALA 

17. 

EFFECT  ON  PHYSICAL  TARGET (8) 

• 

18.  EfTECT  OH  HUMAN  TARGET (8) 

•<Text  "DEV-MUC3-0000"> 

> 

Fr  1  Jun  S  9  4  4  .on  ILi  il (  I  nlm  ioi  <■  tup:  iiui)- 


-Hot - 


Figure  5:  Dual  Window  Emacs-Based  Application  Facility 
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7.  The  Profiling  Facility  and  NLP  System  Profiles 

The  NLP  Evaluation  Tool  is  designed  to  produce  descriptive  profiles  that  describe  NLP 
systems  in  terms  of  the  scores  assigned  for  the  processing  of  specified  linguistic  phenomena. 
The  profiles  are  hierarchically  organized  according  to  the  classification  scheme  discussed 
in  Section  5.  The  profiles  can  be  viewed  or  examined  at  any  level  of  granularity  (levels 
of  granularity  corresponding  to  the  number  of  hierarchy  levels  to  be  displayed).  At  any 
hierarchy  level  N  (N  not  the  bottom  level),  the  scores  from  the  items  or  classes  at  level  N  are 
averaged  to  produce  the  score  for  the  parent  class  or  category  at  level  N+l.  Table  3  shows  a 
sample  system  profile  consisting  of  only  the  top  level  of  the  hierarchy.  The  percentages  are 
the  averages  of  the  scores  produced  for  the  sub-categories  or  sub-items.  The  column  entitled 
“Total  Inputs”  provides  the  number  of  actual  NL  test  inputs  that  were  submitted  to  the 
NLP  system.  Three  test  inputs  were  used  for  each  Tool  item  so  each  entry  in  the  “Total 
Inputs”  column  is  three  times  the  corresponding  entry  in  the  “Total  Items”  column. 


Table  3:  A  System  Profile:  Top  Level  Only 


Category 

Total 

Success 

Correct 

Partial 

Failure 

No  Output 

Items 

Inputs 

# 

wm 

# 

mm 

# 

Hi 

# 

mm 

# 

% 

1 .  Basic  Sentences 

16 

48 

39 

81% 

4 

mm 

2 

mm 

0 

3 

6%~ 

2.  Interrogatives 

9 

27 

26 

wmm 

0 

mm 

0 

K2J| 

0 

mm 

1 

BJ 

3.  Noun  Phrases 

83 

249 

159 

64% 

17 

mm 

8 

3% 

11 

mm 

54 

m*m 

4.  Adverbials 

6 

18 

13 

~72%" 

0 

mm 

0 

mm 

0 

mm 

5 

mm 

5.  Verbs  k  Verb  Phrases 

19 

57 

27 

~47%“ 

17 

0 

Em 

1 

mm 

12 

mm 

6.  Quantifiers 

45 

135 

79 

11 

mm 

4 

3% 

8 

mm 

33 

24% 

7.  Comparatives 

63 

189 

87 

46% 

2 

mm 

11 

~6%r 

6 

mm 

83 

msm 

8.  Connectives 

34 

102 

74 

1z 

2 

mm 

2 

~2%~ 

5 

mm 

19 

■b.i 

9.  Embedded  Sentences 

5 

15 

4 

~2T%~ 

0 

mm 

0 

Em 

0 

mm 

11 

msm 

10.  Reference 

16 

48 

22 

"46%T 

2 

mm 

3 

0% 

5 

mwm 

16 

K 

11.  Ellipsis 

17 

51 

21 

41% 

6 

HI 

2 

mm 

6 

wvm 

16 

Ktl 

12.  Event  Semantics 

39 

117 

37 

wwm 

8 

HI 

3 

3% 

15 

mvm 

54 

46% 

NLP  system  profiles  are  generated  using  the  Microsoft  Excel™  spreadsheet  tool.  The 
Team  designed  a  spreadsheet  using  the  classification  scheme  as  a  basis.  The  scores  assigned 
to  each  item  in  the  Evaluation  Tool  are  entered  into  the  spreadsheet.  The  formulas  embedded 
in  the  spreadsheet  compute  the  averages  for  each  of  the  classes  in  the  hierarchy  from  the 
scores  associated  with  each  element  or  sub-class. 

Graphical  presentations  of  the  evaluation  results  can  also  be  generated  using  the  spread¬ 
sheet  software.  Figure  6  displays  a  graph  of  the  combined  Success  and  Correct  results  of  the 
evaluation  of  System  #1  across  the  four  Interface  Technologists  or  evaluators. 

The  profile  spreadsheet  is  available  from  the  authors. 
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Figure  6:  System  #1  Combined  Success/ Correct  Scores  Across  Evaluators 
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8.  Assessment  of  the  NLP  Evaluation  Tool 

The  purpose  of  the  assessment  exercises  was  to  determine  whether  the  objectives  for  the 
NLP  Evaluation  Tool  were  being  met  and  to  provide  feedback  to  the  developers  to  improve 
and  refine  the  Tool  during  its  development. 

An  assessment  task  (exercise)  was  performed  at  the  end  of  each  of  the  three  six-month 
phases  of  the  Benchmark  I/I  Program.  For  each  of  the  first  two  assessment  tasks,  the  Tool 
was  applied  to  three  different  NLP  systems  by  each  of  three  evaluators,  also  called  Interface 
Technologists.  For  the  final  assessment  task,  the  Tool  was  applied  to  two  different  NLP 
systems  by  each  of  four  evaluators.  Together,  these  three  assessment  exercises  provided  a 
total  of  26  applications  of  the  Tool  to  NLP  systems. 

As  part  of  each  of  the  three  assessment  tasks,  the  Evaluation  Tool  was  evaluated  using 
several  techniques:  statistical  analysis,  item  analysis,  questionnaire,  and  critique  during  use. 
In  this  Final  Report,  we  focus  on  the  third  and  final  assessment  exercise.  The  evaluation 
techniques  used  in  this  final  assessment  exercise  were: 

•  Statistical  disagreement  analysis:  The  data  generated  by  the  application  of  the  Tool 
by  the  four  Interface  Technologists  to  two  NLP  systems  (for  a  total  of  8  applications) 
was  statistically  analyzed  for  consistency  across  Technologists,  although  the  number 
of  subjects  was  small.  The  results  of  this  analysis  are  presented  in  Section  8.3. 

•  Critique  during  use:  The  Interface  Technologists  recorded  problems,  criticisms,  and 
suggestions  regarding  individual  Tool  items  during  their  application  of  the  Tool.  This 
item-by-item  critique  is  too  detailed  for  inclusion  in  this  report. 

•  Scoring  trend  analysis:  The  data  resulting  from  the  application  of  the  Evaluation  Tool 
to  both  of  the  NLP  systems  by  each  of  the  four  Interface  Technologists  was  examined 
for  trends  or  patterns  in  the  scoring  tendencies  of  the  Technologists.  The  results  of  the 
scoring  trend  analysis  is  reported  in  Section  8.4. 

•  Item  analysis:  An  item  analysis  was  performed  to  identify  the  Tool  items  on  which  there 
was  disagreement  among  the  Technologists.  The  copies  of  the  Tool  completed  by  each 
Interface  Technologist  for  each  NLP  system  were  examined  to  determine  the  nature 
and  cause  of  the  disagreements.  Most  of  the  disagreements  were  due  to  Technologist 
errors.  Other  factors  included  ill-defined  evaluation  criteria  and  NLP  system  behavior 
that  was  more  complicated  than  anticipated.  The  results  of  the  item  analysis  are 
discussed  in  Sections  8.5,  8.6,  and  8.7. 

•  Questionnaire:  The  Interface  Technologists  completed  an  Assessment  Questionnaire 
developed  to  evaluate  the  Evaluation  Tool.  The  results  of  this  subjective  assessment 
are  summarized  in  Section  8.8.  Complete  results  are  included  in  Appendix  C. 

The  experimental  design  for  the  assessment  exercises  is  described  in  the  following 
subsection,  with  emphasis  on  the  third  and  final  exercise. 
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8.1  Experimental  Design. 

This  section  describes  the  important  design  features  for  the  assessment  exercises. 

Evaluators  (Interface  Technologists). 

•  To  achieve  an  impartial  assessment  of  the  Tool,  the  Interface  Technologists  had  no 
involvement  in  the  development  of  the  Tool. 

•  To  ensure  that  the  Tool  can  be  used  by  people  who  are  not  trained  linguists,  the 
Interface  Technologists  selected  for  participation  in  the  assessment  exercises  were  were 
somewhat  knowledgable  of  interface  technology,  but  had  little  to  no  formal  education 
in  linguistics,  per  se. 

The  Interface  Technologists  that  served  as  evaluators  for  all  the  assessment  tasks  in¬ 
cluded  one  Calspan  scientist  and  five  graduate  students  at  the  State  University  of  New  York 
(SUNY)  at  Buffalo. 

For  the  third  and  final  assessment  task,  the  four  Interface  Technologists  were  Computer 
Science  graduate  students  at  SUNY  at  Buffalo.  Three  of  the  Interface  Technologists  were 
new  to  the  Tool  and  NLP  systems  and  one  had  participated  during  earlier  phases  and  was 
familiar  with  the  Tool  and  one  of  the  two  NLP  systems. 

NLP  Systems. 

•  To  guard  against  biasing  the  Tool  with  regard  to  a  particular  type  of  NLP  system 
and/or  a  particular  application  domain,  two  types  of  NLP  systems  with  different  ap¬ 
plication  domains  were  used  in  the  assessment  exercises.  One  type  of  system  was  the 
database  front-end  NL  query  system  and  the  other  type  was  the  text  understanding 
system.  The  database  query  systems  used  an  employee  data  base  application  and  the 
text  processing  systems  were  applied  to  extract  information  on  the  MUC-3  terrorism 
domain. 

•  The  NLP  systems  selected  for  use  in  the  assessment  experiments  were  among  the  most 
mature  systems  available  at  the  current  time.  The  selected  NLP  systems  were  a  mix 
of  commercially  available  systems  and  advanced  research  products.  The  intent  was  to 
use  systems  that  provide  a  rich  suite  of  NLP  capabilities. 

•  To  minimize  bias  with  regard  to  particular  NLP  systems,  at  least  one  of  the  NLP 
systems  used  at  each  assessment  milestone  was  a  system  that  had  not  been  used  in  the 
previous  assessment  activities. 

Over  the  course  of  the  entire  project,  the  NLP  Evaluation  Tool  was  used  with  three 
commercial  “off  the  shelf”  (COTS)  natural  language  database  query  front  end  systems  and 
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three  advanced  research  prototypes  that  were  implemented  for  the  MUC-3  evaluation  exer¬ 
cise. 

For  the  last  (third)  phase  assessment  exercise,  we  used  a  COTS  database  query  front 
end  system  and  one  of  the  advanced  research  prototypes  implemented  for  MUC-3. 

Tool  Application  Procedure. 

•  To  minimize  bias  due  to  order  of  Tool  application  by  the  Interface  Technologists,  a 
Latin  square  design  was  used  for  the  Tool  applications.  This  Latin  square  design  is 
illustrated  in  Table  4. 


Table  4:  Latin  Square  Design  for  the  Assessment  Task 


Phase 

Evaluator 

NLP  System 

I 

IT  #1 

SYS#1 

SYS#2 

SYS#3 

I 

IT  #2 

SYS#2 

SYS#3 

SYS#1 

I 

IT  #3 

SYS#3 

SYS#1 

SYS#2 

II 

IT  #1 

SYS#1 

SYS#2 

SYS#4 

II 

IT  #2 

SYS#2 

SYS#4 

SYS#1 

II 

IT  #4 

SYS#4 

SYS#1 

SYS#2 

III 

IT  #1 

SYS#1 

SYS#5 

III 

IT  #5 

SYS#1 

SYS#5 

III 

IT  #6 

SYS#5 

SYS#1 

III 

IT  #7 

SYS#5 

SYS#1 

•  To  avoid  confusion  between  NLP  systems,  each  Interface  Technologist  applied  the  com¬ 
plete  Tool  to  one  NLP  system  and  then,  when  finished  with  the  one  system,  proceeded 
to  apply  the  Tool  to  another.  The  Interface  Technologists  did  not  switch  back  and 
forth  between  systems  while  applying  the  Tool. 

•  For  the  third  and  final  assessment  exercise,  each  Technologist  used  three  test  inputs 
for  each  evaluation  Tool  item  in  an  attempt  to  maximize  the  representativeness  of  the 
results  with  regard  to  the  systems’  capabilities.  The  NLP  system’s  response  to  each  of 
the  three  test  inputs  was  scored  individually  by  the  Technologist. 


8.2  Training 

The  Team  developed  a  short  training  program  and  training  materials  to  familiarize  the 
Interface  Technologists  with  the  evaluation  Tool,  the  NLP  systems,  and  the  Emacs-based 
dual-window  facility.  Some  of  the  training  materials  also  served  as  reference  aids  when 
the  Interface  Technologists  began  the  task  of  formally  applying  the  Tool  to  the  NLP  sys¬ 
tems.  Training  was  generally  conducted  over  a  two-day  period  and  included  instruction  find 
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hands-on  practice  in  applying  the  evaluation  Tool  to  the  two  NLP  systems.  Each  Interface 
Technologist  was  initially  trained  on  the  NLP  system  that  he/she  used  first,  and,  after  finish¬ 
ing  the  evaluation  of  the  first  NLP  system,  he/she  was  provided  with  training  on  the  second 
NLP  system  just  before  beginning  formal  application  of  the  Tool  to  the  second  system. 

The  following  types  of  training  materials  were  used.  Copies  are  available  from  the 

authors. 

1.  Training  Objectives  and  Materials 

2.  Training  Outline 

3.  Overview  of  Purpose,  Philosophy,  and  Objectives  of  the  Evaluation  Tool 

4.  Introduction  to  The  Evaluation  Tool 

5.  Evaluation  Tool  Procedures 

6.  Guide  to  Notation 

7.  Emacs  Instructions 

8.  The  NLP  System  Procedures 

9.  Using  the  NLP  System 

10.  The  NLP  System  Application  Domain 


The  following  is  a  summary  of  the  guidelines  that  the  Interface  Technologists  were 
instructed  use: 

•  Take  your  time  and  apply  each  item  thoughtfully  (no  time  limits) 

•  Take  breaks  of  5  to  10  minutes  each  hour  (more  if  tired) 

•  Submit  multiple  test  inputs  for  each  test  item  (record  three) 

•  Use  variety  in  test  inputs  (within  allowed  specifications)  for  each  Tool  item 

•  Carefully  consider  the  NLP  system  output/response  against  evaluation  criteria 

•  Record  a  score  for  each  submitted  test  input 

•  Build  on  previously  used  language  patterns  processed  successfully  by  the  system  (do 
not  re-use  failed  linguistic  phenomena) 

•  Make  notes  and  comments 

—  Clarify  scores,  if  necessary 
-  Note  strengths  and  weakness  of  the  Tool 

•  Do  not  fault  the  NLP  system  when  its  vocabulary  has  been  exceeded  (unless  it  is 
central  to  the  phenomenon  under  test) 
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•  Ask  for  help  if  problems  occur  or  directions/instructions  are  needed 


8.3  Reliability /Agreement  Data  Analysis 

Data  analysis  for  agreement  or  consistency  across  evaluators  required  the  following  major 
tasks: 

•  Determine  a  definition  of  a  match  or  agreement  across  evaluators  (on  a  per  item  basis). 

•  For  each  NLP  system  and  for  each  Evaluation  Tool  item,  apply  the  match  definition 
to  determine  whether  the  scores  of  the  four  evaluators  were  in  agreement. 

•  Using  the  Binomial  Test  for  statistical  significance,  determine  which  sections  and  sub¬ 
sections  produced  reliable  results.  The  Binomial  Test  was  used  since  there  are  two 
possible  outcomes  (match  or  non-match)  for  each  “trial”  or  evaluation  item. 

This  approach  to  assessing  reliability  has  some  similarity  to  the  reliability  study  per¬ 
formed  by  Hix  and  Schulman  for  their  method  of  evaluating  human-computer  interface  de¬ 
velopment  tools  [Hix91]. 

These  tasks  are  discussed  in  the  following  subsections. 

8.3.1  Measuring  Agreement 

The  selection  of  the  algorithm  for  measuring  agreement  among  the  Interface  Technologists 
involved  consideration  of  numerous  candidate  approaches.  The  selection  was  made  by  com¬ 
paring  the  results  of  each  of  the  candidate  approaches  to  the  combined  results  when  three 
of  the  Evaluation  Tool  designers,  working  independently,  applied  their  own  heuristics  and 
engineering  judgement  to  a  small  sample  (50)  of  possible  score  combinations.  Based  on  this 
comparison  study,  one  approach  for  measuring  whether  the  four  technologists  agreed  in  their 
assessments  was  selected. 

The  candidate  approaches  can  be  classified  in  three  categories:  (1)  combined  spread 
methods;  (2)  raw  pattern  methods;  and  (3)  numerical  methods.  The  combined  spread  meth¬ 
ods  involve  two  steps:  first,  combine  the  three  scores  per  Evaluation  Tool  item  obtained  from 
each  Interface  Technologist  into  a  single  composite  score  per  Tool  item;  and  then  compare 
the  degree  of  agreement  of  the  composite  scores  across  technologists.  The  raw  pattern  meth¬ 
ods  involve  direct  comparisons  across  all  scores  obtained  by  the  four  Interface  Technologists. 
The  degree  to  which  all  the  scores  line  up  under  one  or  a  cluster  of  scores  defines  the  degree 
of  agreement.  The  numerical  approaches  involve  awarding  points  based  on  scores  obtained 
by  the  Interface  Technologists,  weighted  by  frequency  of  occurrence,  and  assessing  agreement 
between  subjects  mathematically. 
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The  approach  that  was  found  to  best  align  with  the  results  of  the  Evaluation  Tool 
designers  was  from  the  numerical  method  category.  This  method  is  discussed  in  the  following 
paragraphs. 

This  approach  consists  of  two  major  steps: 

1.  Calculate  a  combined  numerical  score  (weighted  average)  for  the  three  letter  scores 
generated  for  each  Tool  item.  (Recall  from  Section  6.2  that  the  possible  letter  scores 
are  S,C,P,F,  and  N.)  That  is,  the  Technologist  used  three  test  inputs  for  each  Tool 
item  and  assigned  the  NLP  system  a  letter  score  for  each  of  its  three  corresponding 
responses. 

2.  For  each  NLP  system  and  for  each  Evaluation  Tool  item,  determine  whether  the  numer¬ 
ical  scores  of  the  four  evaluators  agree  (match)  based  on  the  selected  match  definition. 

Computing  combined  numerical  scores. 

The  following  paragraphs  describe  the  method  used  to  calculate  a  combined  numerical  score 
(weighted  average)  for  each  evaluator  for  each  Tool  item.  First,  a  numerical  score  was 
assigned  to  each  possible  letter  score: 

•  S  =  20  (or  21  if  two  S  scores  obtained  by  a  subject) 

•  C  =  15, 

•  P  =  10, 

•  F  =  5,  and 

•  N  =  5. 

When  two  or  more  S  scores  were  obtained  by  an  Interface  Technologist,  21  points  were  used 
rather  that  20  because  it  was  felt  that  a  success  should  be  harder  to  match  than  other 
scores.  Furthermore,  this  approach  seemed  to  agree  with  the  empirical  studies  performed  by 
the  developers  when  determining/designing  the  score  combining  method. 

A  weighted  average  was  then  calculated  for  each  Interface  Technologist  based  on  the 
three  scores  given  for  each  Evaluation  Tool  test  item.  The  method  for  computing  the 
weighted  average  was: 

•  If  the  three  scores  are  the  same  or  if  the  three  scores  are  all  different,  then  compute  an 
average  with  the  scores  weighted  equally. 

•  If  there  are  two  identical  scores  and  one  score  that  is  different,  then  use  weights  of  .60 
and  .40,  respectively,  for  the  two  score  values  when  computing  the  weighted  average. 
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The  reason  for  using  a  60-40  split  in  the  case  where  two  scores  are  the  same  and  one  is 
different  (rather  than  a  67-33  split),  is  because  with  three  inputs  scored,  it  is  not  possible  to 
tell  whether  the  theoretical  likelihood  of  obtaining  one  or  the  other  score  is  50-50  or  67-33. 
Using  a  60-40  split  accommodates  both  possibilities  while  favoring  the  actual  data  obtained. 

The  following  are  three  examples: 


•  Scores  of  S,C,and  P  result  in  a  weighted  average  of  (20  +  15  +  10)  /  3  =  15 

•  Scores  of  C,C,and  P  result  in  a  weighted  average  of  .6(15)  +  .4(10)  =  13 

•  Scores  of  S,S,and  P  result  in  a  weighted  average  of  .6(21)  -f  -4(10)  =  16.6 

Since  there  are  five  (5)  scores  possible  on  each  test  item  and  each  Interface  Technologist 
submits  3  scores,  there  are  35  possible  score  combinations.  These  are  shown  in  Table  5  below. 
The  combined  numerical  score  (weighted  average)  is  also  shown  for  each  case. 

Table  5.  Possible  Score  Combinations 


S: 

C: 

P: 

F: 

N: 

Combined 

Score: 

3.00 

0.00 

0.00 

0.00 

0.00 

21.00 

2.00 

1.00 

0.00 

0.00 

0.00 

19.00 

2.00 

0.00 

1.00 

0.00 

0.00 

17.00 

2.00 

0.00 

0.00 

1.00 

0.00 

15.00 

2.00 

0.00 

0.00 

0.00 

1.00 

15.00 

0.00 

3.00 

0.00 

0.00 

0.00 

15.00 

1.00 

2.00 

0.00 

0.00 

0.00 

17.00 

0.00 

2.00 

1.00 

0.00 

0.00 

13.00 

0.00 

2.00 

0.00 

1.00 

0.00 

11.00 

0.00 

2.00 

0.00 

0.00 

1.00 

11.00 

0.00 

0.00 

3.00 

0.00 

0.00 

10.00 

1.00 

0.00 

2.00 

0.00 

0.00 

14.00 

0.00 

1.00 

2.00 

0.00 

0.00 

12.00 

0.00 

0.00 

2.00 

1.00 

0.00 

8.00 

0.00 

0.00 

2.00 

0.00 

1.00 

8.00 

0.00 

0.00 

0.00 

3.00 

0.00 

5.00 

1.00 

0.00 

0.00 

2.00 

0.00 

11.00 

0.00 

1.00 

0.00 

2.00 

0.00 

9.00 

0.00 

0.00 

1.00 

2.00 

0.00 

7.00 

0.00 

0.00 

0.00 

2.00 

1.00 

5.00 

0.00 

0.00 

0.00 

0.00 

3.00 

5.00 
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1.00 

0.00 

0.00 

0.00 

2.00 

11.00 

0.00 

1.00 

0.00 

0.00 

2.00 

9.00 

0.00 

0.00 

1.00 

0.00 

2.00 

7.00 

0.00 

0.00 

0.00 

1.00 

2.00 

5.00 

1.00 

1.00 

1.00 

0.00 

0.00 

15.00 

1.00 

1.00 

0.00 

1.00 

0.00 

13.33 

1.00 

1.00 

0.00 

0.00 

1.00 

13.33 

1.00 

0.00 

1.00 

1.00 

0.00 

11.67 

1.00 

0.00 

1.00 

0.00 

1.00 

11.67 

1.00 

0.00 

0.00 

1.00 

1.00 

11.00 

0.00 

1.00 

1.00 

1.00 

0.00 

10.00 

0.00 

1.00 

1.00 

0.00 

1.00 

10.00 

0.0^ 

1.00 

0.00 

1.00 

1.00 

9.00 

0.00 

0.00 

1.00 

1.00 

1.00 

7.00 

Determining  agreement. 

The  Match  Definition  used  to  determine  agreement  across  the  four  Interface  Technologists 
on  a  per  item  basis  is  a  function  of  the  spread  between  the  maximum  and  minimum  of  the 
four  numerical  scores  for  the  Technologists.  The  match  definition  is: 


•  If  (MaximumScore  -  MinumumScore)  <  8,  then  the  four  scores  match; 

•  Otherwise,  the  four  scores  do  not  match. 

This  criteria  was  applied  to  the  Technologists’  scores  for  each  Tool  item  for  both  NLP 
systems.  The  following  table  shows  two  examples. 

Table  6:  An  Example  Set  of  Matching  Scores  and  a  Non-Matching  Set 


NO  MATCH 

MATCH 

Technologist  #1:  21 
Technologist  #2:  11 
Technologist  #3:  15 
Technologist  #4:  12 

Technologist  #1:  15 
Technologist  #2:  17 
Technologist  #3:  17 
Technologist  #4:  11 

Max  -  Min  =  21-11  =  10 

Max  -  Min  =  17-11  =  6 

8.3.2  Statistical  Analysis 

After  applying  the  match  definition  to  the  Technologists’  scores  to  determine  which  Eval¬ 
uation  Tool  items  resulted  in  agreement  across  the  Interface  Technologists,  the  next  step 
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was  to  determine  whether  the  resulting  level  of  agreement  was  statistically  significant  (i.e., 
significantly  more  than  would  be  expected  by  chance  alone).  This  was  accomplished  on  a 
section-by-section  basis  using  the  Binomial  Test.  Since  there  is  a  finite  probability  of  obtain¬ 
ing  a  match  across  subjects  (using  the  approach  described  above)  it  is  possible  to  calculate 
the  likelihood  of  obtaining  the  number  of  matches  obtained  in  each  section  by  chance  alone. 
This  is  done  using  the  Binomial  Probability  Distribution.  The  method  for  calculating  the 
probability  of  obtaining  X  or  more  matches  in  a  section  containing  N  items  is  shown  below: 


The  probability  of  X  or  more  matches  in  a  section  containing  N  items  = 

sfL*  (  lN  )  p'(i  •  p)w-' 

where  (  (  j  =  N!  /  [  I!  (N-I)!j  and 
N  =  Number  of  test  items 

p  =  Probability  of  obtaining  a  match  on  each  item  by  chance 


A  computer  program  was  written  in  the  C  programming  language  to  compute  the 
probability  p  of  obtaining  a  match  on  any  arbitrary  Tool  item  by  chance  alone.  The  program 
essentially  iterates  through  all  possible  letter  score  combinations  and  corresponding  weighted 
averages  (see  Table  5)  for  the  four  Technologists  and  identifies  and  counts  the  number  of 
combinations  that  satisfy  the  match  criteria.  The  program  also  computes  the  total  number 
of  combinations.  The  final  step  of  the  program  yields  the  probability  p  of  a  match  on  an 
item  by  chance  by  dividing  the  total  number  of  letter  score  combinations  that  satisfied  the 
match  criteria  by  the  total  count  of  all  letter  score  combinations.  This  program  is  defined 
using  the  following  algorithm. 

ALGORITHM: 

1.  Enumerate  ways  of  combining  S,  C,  P,  F,  and  F  in  an  array  called  Scores[35][6],  where: 

•  35  =  the  number  of  possible  score  combinations  given  5  possible  columns  (S,  C, 
P,  F,  N)  in  which  to  place  a  total  of  3  results.  The  order  of  the  3  results  does  not 
matter,  so  a  score  containing  one  S  and  two  Cs  is  not  distinct  from  a  score  made 
up  of  one  C,  one  S,  and  one  C. 

•  6  =  the  five  columns  S,  C,  P,  F,  and  N,  plus  a  sixth  column  to  hold  the  corre¬ 
sponding  combined  numerical  score  (weighted  average). 

2.  Define  an  array  called  Points[5],  where  the  numerical  scores  20,  15,  10,  5,  5  are  assigned 
to  the  columns  S,  C,  P,  F,  and  N,  respectively. 
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3.  Compute  the  value  for  each  of  the  35  possible  score  combinations  in  the  Scores  array 
(see  Table  5). 

4.  Compute  the  agreement  probability: 

Given  4  Technologists  and  the  35  possible  numerical  combined  scores  (weighted  aver¬ 
ages),  there  are  1,500,265  (354  =  1,500,265)  possible  ways  of  combining  the  possible 
numerical  combined  scores  (weighted  averages).  We  use  four  nested  loops,  one  for 
each  Technologist,  and  each  loop  variable  indexes  through  the  35  possible  numerical 
combined  scores  (weighted  averages)  in  the  Scores  array  for  the  particular  Technologist. 

For  the  innermost  nested  loop,  the  loop  variable  indexes  through  all  possible  combined 
scores  (weighted  averages)  for  the  one  Technologist,  holding  the  combined  scores  for 
the  other  Technologists  constant.  Each  next  outer  loop  cycles  through  the  combined 
scores  for  the  other  Technologists.  Thus  the  program  considers  and  counts  the  total 
number  of  possible  agreements  given  every  permutation  of  combined  scores  (weighted 
averages)  for  the  4  Technologists. 

The  code  follows  the  following  basic  structure: 

For  i  =  1  to  35  do 

For  j  =  1  to  35  do 

For  k  =  1  to  35  do 
For  1  =  1  to  35  do 

Add  1  to  the  Total  Number  of  Permutations. 

If  the  4  scores  represented  by  the  counters  i,  j,  k,  1  agree, 
that  is,  if  (MaximumScore  -  MinimumScore)  <  8, 
then  add  1  to  the  Number  of  Agreements. 

End  For-loop 
End  For-loop 
End  For-loop 
End  For-loop 

At  the  end  of  the  looping  compute: 

Agreement  Probability  =  (Number  of  Agreements)  /  (Total  Number  of  Permutations) 

Agreement  results  were  computed  for  all  sections  and  subsections  of  the  Tool.  Table 
7  presents  the  results  of  the  reliability  study  for  the  twelve  major  sections  of  the  Tool.  The 
results  for  all  the  Tool  subsections  are  not  presented  in  this  report  due  to  the  voluminous 
nature  of  the  results,  but  are  available  from  the  authors. 

Table  7  shows  the  results  for  each  of  the  two  NLP  systems.  The  table  shows  the  number 
of  agreements  per  section,  the  likelihood  of  this  number  of  agreements  occurring  by  chance 
based  on  the  Binomial  Test,  and  the  percentage  of  agreements.  The  rightmost  two  columns 
indicate  whether  the  number  of  agreements  per  section  for  each  of  the  two  NLP  systems  was 


Table  7:  Reliability/ Agreement  Study  Results  for  the  Major  Sections 
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to  significance 

statistically  significant  using  a  0.05  level  of  significance.  The  “YES”  or  “NO”  indicate  that 
the  number  of  agreements  was  significant  or  not  significant,  respectively,  while  the  asterisks 
flag  the  sections  that  were  close  to  significant. 

In  order  to  understand  the  reasons  for  the  lack  of  a  significant  level  of  agreement  in 
some  of  the  Tool  sections,  we  studied  the  data  from  the  Tool  applications  of  each  of  the  four 
Interface  Technologists  to  each  of  the  two  NLP  systems.  This  resulted  in  two  products.  The 
first  study  involved  a  trend  analysis  and  the  second  involved  an  error  analysis. 

8.4  Scoring  Trend  Analysis 

A  study  was  undertaken  to  examine  the  data  for  trends  or  patterns  in  the  tendencies  of 
the  Interface  Technologists.  The  results  of  this  trend  analysis  are  shown  in  Table  8.  The 
table  shows  that  Interface  Technologist  #4  had  a  tendency  to  assign  very  low  scores  to 
NLP  System  #1  with  disproportionate  frequency  in  comparison  to  the  other  three  Interface 
Technologists.  The  table  also  shows  that  Interface  Technologist  #4  had  a  tendency  to  assign 
very  high  scores  to  NLP  System  #2  with  disproportionate  frequency  in  comparison  to  the 
other  three  Interface  Technologists.  Overall,  Interface  Technologist  #4  had  almost  twice 
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Table  8:  Results  of  the  Scoring  Trend  Analysis 

Technologist  Scoring 
Trend  Analysis 
frequency  (%  of  total  items) 


Single 

Highest 

Single 

Lowest 

Total 

Scores 

Scores 

Extrema: 

System  1: 
Technologist  1 

40  (27.78) 

31  (16.85) 

71  (21.65) 

Technologist  2 

36  (25.00) 

42  (22.83) 

78  (23.78) 

Technologist  3 

31  (21.53) 

33(17.93) 

64(19.51) 

Technologist  4 

37  (25.69) 

78  (42.39) 

115(35.06) 

Totals: 

144 

184 

328 

System  2: 

Technologist  1 

25  (10.87) 

48  (35.29) 

73(19.95) 

Technologist  2 

50(21.74) 

21  (15.44) 

71  (19.40) 

Technologist  3 

52(22.61) 

32  (23.53) 

84  (22.95) 

Technologist  4 

103(44.78) 

35  (25.74) 

138  (37.70) 

Totals: 

230 

136 

366 

Overall: 

Technologist  1 

65(17.38) 

79  (24.69) 

144  (20.75) 

Technologist  2 

86  (22.99) 

63(19.69) 

149(21.47) 

Technologist  3 

83(22.19) 

65  (20.31) 

148(21.33) 

Technologist  4 

140  (37.43) 

113(35.31) 

253  (36.46) 

Grand  Totals: 

374 

320 

694 

the  number  of  extreme  scores  causing  non-matches  to  occur.  The  reasons  for  this  trend  in 
scoring  on  the  part  of  Interface  Technologist  #4  are  not  clear.  The  reasons  could  include: 
(1)  inadequate  training  to  the  Tool  and  to  the  NLP  systems,  (2)  partial  misunderstanding 
of  the  score  definitions  and  their  applicability  rules,  and/or  (3)  a  natural  tendency  to  grade 
at  the  extremes.  Our  debriefing  of  this  Interface  Technologist  indicated  that  there  was  some 
misunderstanding  of  the  score  definitions,  which  means  that  corrective  action  should  focus  on 
items  (I)  and  (2),  above,  that  is,  improved  training  and  providing  more  explicit  definitions 
of  the  scores. 
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8.5  Evaluator  Errors 


We  conducted  an  analysis  of  the  errors  committed  by  the  Interface  Technologists  that  con¬ 
tributed  to  the  lack  of  consistency  across  technologists  in  the  assessment  exercise.  The  errors 
were  categorized  and  the  following  list  displays  the  error  types  in  order  of  frequency  of  oc¬ 
currence,  with  the  most  frequent  listed  first.  Following  the  list  of  evaluator  error  types,  each 
error  type  is  discussed  briefly  and  examples  are  provided  for  illustrative  purposes. 

1.  Lack  of  variety  in  test  inputs 

2.  Incorrect  or  inappropriate  use  of  scores 

3.  Lack  of  understanding  of  scores 

4.  Misunderstanding  of  evaluation  criteria 

5.  Test  input  includes  a  failed  linguistic  phenomenon 

6.  Test  input  that  is  outside  of  the  system’s  application  domain 

7.  Test  input  that  does  not  test  the  described  linguistic  capability 

8.  Grammar  or  spelling  mistakes 

9.  Inability  to  compose  test  input 


1.  Lack  of  variety  in  test  inputs 

In  cases  where  an  evaluator  used  very  similar  inputs  for  the  three  items  in  a  test,  the  score 
would  be  erroneously  high  or  low  with  respect  to  the  system’s  behavior  and  with  respect  to 
the  other  evaluators’  scores. 


•  Lack  of  variety  with  respect  to  Syntax. 

Example:  When  testing  the  database  natural  language  query  front-end,  one  of  the 
evaluators  input  the  following  three  sentential  test  inputs  when  testing  non-assertive 
comparatives  with  adjective  gapping  (Subsection  1.3.2  of  Section  VH): 


“List 

“List 

“List 


an  employee  who  is 
an  employee  who  is 
an  employee  who  is 


not  so  experienced 
not  so  old  as  John 
not  so  well  paid  as 


as  John  Tower  is.” 
Tower  is.” 

John  Tower  is.” 
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The  system  could  not  properly  parse  the  input  and  gave  very  poor  output,  causing  the 
evaluator  to  assign  a  score  of  “P”.  Another  evaluator,  however,  tried  the  alternative 
recommended  wording  which  was  presented  in  the  evaluation  Tool  in  this  item,  namely, 
to  use  “not  as  ...  as”  in  place  of  “not  so  ...  as”.  When  this  is  done,  the  system  succeeds 
on  the  above  inputs.  Since  the  two  wordings  were  given  as  alternatives,  he  evaluator 
'hould  have  tried  the  second  option  when  the  first  failed. 

•  Lack  of  variety  with  respect  to  Semantics. 

Example:  In  each  of  his  test  inputs  for  the  database  natural  language  query  front-end, 
one  of  the  evaluators  made  reference  to  or  asked  about  the  location  of  the  residence  of 
employees,  although  the  system  told  the  evaluator  each  time  that  it  had  no  information 
in  its  database  on  the  topic  of  where  employees  live.  The  evaluator  could  easily  have 
found  test  inputs  within  the  scope  of  the  database.  The  evaluator  gave  the  system  an 
N  (failing)  score  for  each  output. 


2.  Incorrect  or  inappropriate  use  of  scores 

Evaluators  occasionally  scored  systems  incorrectly  (e.g.,  assigned  an  F  to  the  system  when 
the  system’s  response  was  correct)  and  evaluators  occasionally  scored  systems  too  liberally 
or  too  harshly  (e.g.,  an  evaluator  would  assign  a  score  of  S  when  the  system  did  not  fully 
satisfy  the  evaluation  criteria  for  the  test  item). 

Example:  An  example  of  an  inappropriate  score  assignment  was  the  assignment  of  a  score 
of  S  that  was  too  liberal.  This  occurred  when  the  criteria  was  not  applied  stringently  enough. 
For  example,  in  Section  III  on  Noun  Phrases,  5.2,  one  of  the  evaluators  assigned  a  score  of 
S  to  the  system  for  its  response  to  the  input: 

“Terrorists  kidnapped  the  son  of  the  governor.” 

even  though  the  criteria  states: 

Criteria:  Demonstrated  understanding  of  the  relationship  between  the  noun  phrase  being 
postmodified  and  the  postmodifying  of-phrase. 

The  template  produced  does  not  demonstrate  that  the  system  satisfied  the  criteria,  namely, 
it  shows  no  relation  between  the  noun  phrase  being  postmodified  and  the  postmodifying 
of- phi  ase.  Instead,  the  template  simply  repeats  the  phrase. 

11.  HUMAN  TARGET:  ID(S)  "son  of  the  governor" 

12.  HUMAN  TARGET:  TOTAL  NUM  1 

13.  HUMAN  TARGET:  TYPE(S)  CIVILIAN:  "son  of  the  governor" 

Example:  One  of  the  evaluators  assigned  an  F  when  the  system  processed  the  input  cor¬ 

rectly.  This  occurred  in  Section  VI  on  Quantifiers  when  testing  the  system’s  processing  of 
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ordinal  number  quantifiers.  The  evaluator  used  the  following  test  input,  which  the  system 
processed  correctly. 

“List  the  first  woman  employee  in  the  New  York  branch.” 

The  system  responded  with  an  acceptable  statement  of  what  was  requested  and  presented 
information  on  the  woman  in  table  form,  as  follows: 

The  earliest  female  salesperson  that  works  for  the 
New  York  City  branch  and  her  start  date. 

first  name  I  last  name  I  year  started 

Jennifer  J  ort  1  1988 


3.  Lack  of  understanding  of  scores 

Evaluators  occasionally  assigned  scores  to  system  outputs  that  indicated  that  they  did  not 
understand  the  definitions  of  the  scoring  choices  (i.e.,  S,  C,  P,  F,  N)  or  the  conditions  under 
which  they  were  to  be  applied.  (Refer  back  to  Section  6  for  the  score  definitions.) 

Example:  In  Section  IV,  Subsection  1.1.2  on  PPs  as  Adverbials,  three  evaluators  each  used 
the  test  input  sentence  (which  was  offered  as  an  example  by  the  Tool): 

“In  Guatemala  City,  the  guerrillas  bombed  a  bank.” 

The  criteria  for  this  Tool  item  is: 


Criteria:  Demonstrated  understanding  of  the  meaning  expressed  by  the  prepositional  phrase 
in  initial  position. 


The 

template  produced  by  the  system 

is  shown  here: 

0. 

MESSAGE  ID 

DEV-MUC3-0000 

1. 

TEMPLATE  ID 

1 

2. 

DATE  OF  INCIDENT 

-  31  OCT  89 

3. 

TYPE  OF  INCIDENT 

BOMBING 

4. 

CATEGORY  OF  INCIDENT 

TERRORIST  ACT 

5. 

PERPETRATOR:  ID  OF  INDIV(S) 

"guerrillas" 

6. 

PERPETRATOR:  ID  OF  ORG(S) 

- 
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7.  PERPETRATOR:  CONFIDENCE 

8.  PHYSICAL  TARGET:  ID(S) 

9.  PHYSICAL  TARGET:  TOTAL  NUM 

10.  PHYSICAL  TARGET:  TYPE(S) 

11.  HUMAN  TARGET:  ID(S) 

12.  HUMAN  TARGET:  TOTAL  NUM 

13.  HUMAN  TARGET:  TYPE(S) 

14.  TARGET:  FOREIGN  NATION(S) 

15.  INSTRUMENT:  TYPE(S) 

16.  LOCATION  OF  INCIDENT 

17.  EFFECT  ON  PHYSICAL  TARGET (S) 

18.  EFFECT  ON  HUMAN  TARGET(S) 


"bank" 

1 

FINANCIAL:  "bank" 


* 

GUATEMALA 


The  scores  assigned  by  the  three  evaluators  for  this  result  were  F,  P,  and  S.  It  is  clear  that 
the  system  at  least  partially  understood  the  prepositional  phrase,  but  it  has  lost  some  of  the 
locational  information  and  the  criteria  is  not  completely  satisfied. 

Certainly  an  S  is  too  accepting  of  the  criteria,  ;>.id  certainly  the  F  is  too  severe.  Since  the 
system’s  output  is  correct  except  for  the  loss  of  specificity  in  the  locative  information,  a  C 
score  should  have  been  assigned.  Any  score  less  than  a  C  is  an  incorrect  score. 


4.  Misunderstanding  of  evaluation  criteria 

Every  Tool  item  includes  a  statement  of  an  evaluation  criteria  designed  especially  for  the 
particular  linguistic  phenomenon  being  testing  in  the  item.  On  occasion,  an  evaluator  seemed 
to  misunderstand  an  evaluation  criteria. 

Example:  In  Section  I,  Subsection  1.1,  the  criteria  is  worded: 


Criteria:  Demonstrated  understanding  by  producing  an  acceptable  paraphrase. 


During  training,  the  evaluators  were  informed  that  the  templates  produced  by  the  text 
processing  system  would  be  accepted  as  paraphrases,  whenever  the  term  “paraphrases”  was 
used.  However,  one  of  the  evaluators  gave  scores  of  C  to  all  the  test  items  in  the  subsection, 
and  wrote:  “System  does  not  produce  paraphrases.  This  test  is  not  applicable”.  The  system 
was  indeed  producing  correct  output  and  the  output  should  have  been  scored  as  S,  but  the 
evaluator  misunderstood  the  term  “paraphrase.” 

5.  Test  input  includes  a  failed  linguistic  phenomenon 

This  means  that  the  evaluator  entered  a  test  input  that  violates  the  fundamental  rule  that 
each  test  input  should  consist  solely  of  linguistic  phenomena  upon  which  the  system  already 
succeeded,  except  for  the  linguistic  phenomenon  being  tested.  Test  inputs  should  never 
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include  any  phenomena  on  which  the  NLP  system  has  already  failed. 

Example:  When  testing  comparative  adjective  phrases  in  Section  VII,  one  of  the  evaluators 
entered  “list  an  employee  who  is  more  experienced  than  Carl  Frank  is.”  The  system  pro¬ 
cessed  the  comparative  adjective  phase  correctly  and  responded  with  correct  output,  but  the 
evaluator  gave  the  system  a  poor  score  because  the  system  did  not  respond  with  information 
on  just  one  employee  (“an  employee”). 

Example:  When  testing  embedded  sentences  with  an  embedded  yes-no  interrogative  in 
Section  IX,  Subsection  3,  one  of  the  evaluators  entered  an  imperative  using  the  word  “state,” 
namely 

“State  whether  John  Tower  works  in  the  New  York  branch.” 

However,  the  NLP  system  does  not  recognize  the  verb  “state.”  The  system  succeeds  on  the 
input  if  the  word  “state”  is  changed  to  “tell  me.” 

Example:  One  of  the  evaluators  entered  inappropriate  declarative  sentences  to  the  database 
query  system  many  times  throughout  the  Tool  application  when  declarative  sentences  were 
not  the  test  focus  and  the  system  had  already  demonstrated  that  could  only  process  limited 
types  of  declarative  sentences. 

Example:  A  example  of  this  occurred  when  some  of  the  evaluators  used  the  phrase  “the 
government  house”  as  a  target  of  terrorist  activity  when  testing  the  MUC-3  text  processing 
system.  In  spite  of  MUC  rules  to  the  contrary,  the  system  being  tested  did  not  accept 
this  phrase  in  object  position,  because  of  a  typographical  error  in  its  own  knowledge  base. 
Evaluator  #3  discovered  that  the  misspelling  “goverment  house”  was  accepted  by  the  system 
and  used  it.  The  other  evaluators  continued  to  use  the  correctly  spelled  version  in  subsequent 
test  inputs,  in  spite  of  the  system’s  having  previously  failed  on  the  phrase. 

6.  Test  input  that  is  outside  of  the  system’s  application  domain 

Another  type  of  error  that  committed  by  one  of  the  evaluators  was  to  submit  test  input  to 
the  NLP  system  that  was  not  within  the  application  domain  of  the  system.  One  of  the  rules 
for  applying  the  Evaluation  Tool  is  that  all  test  inputs  for  the  NLP  systems  be  within  their 
domains.  Furthermore,  if  a  test  input  is  inadvertently  used  that  violates  this  rule,  then  that 
test  should  be  discarded  (no  score  assigned)  and  the  system  should  be  re-tested  using  new 
input.  Typically,  the  interface  technologist  should  immediately  be  able  to  determine  the 
admissibility  of  the  input  from  the  system’s  response. 

Example:  When  testing  the  database  query  front-end  system:  one  of  the  evaluators  asked 
questions  about  the  “degrees”  of  the  employees,  but  the  database  has  no  information  on  the 
degrees  of  the  employees  and  responded  by  stating  this  fact. 

Example:  When  testing  the  NL  database  query  system,  one  of  the  evaluators  asked  ques¬ 
tions  about  or  made  reference  to  the  place  of  residence  for  employees  (e.g.,  “does  John 
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Tower  live  in  New  York”  or  “employees  who  live  in  New  York”),  but  the  database  has  no 
information  on  where  employees  live  and  responded  by  stating  this  is  the  case. 

7.  Test  input  that  does  not  test  the  described  linguistic  capability 

Each  Tool  item  is  designed  to  test  a  particular  linguistic  capability.  One  type  of  evaluator 
error  consisted  of  the  evaluator  composing  a  test  input  for  a  Tool  item  such  that  the  test 
input  did  not  test  the  particular  linguistic  capability  identified  in  the  Tool  item. 

Example:  In  the  section  on  verb  tenses,  when  testing  future  tense  the  evaluator  enters  an 
input  using  past  tense. 

Example:  In  Section  III  Noun  Phrases,  Subsection  5. 1.2. 6,  when  testing  a  partitive  genitive, 
one  of  the  evaluators  used  the  following  item  which  contains  a  possessive  genitive,  instead: 

“The  city’s  public  buses  were  burned  by  the  terrorists,” 


8.  Grammar  or  spelling  mistakes 

Occasionally  an  evaluator  used  a  test  input  that  was  ungrammatical  or  contained  a  spelling 
error. 

Example:  In  Section  III  Noun  Phrases,  Subsection  4.6  (d)  Premodifier  Combinations,  one 
of  the  evaluators  used  the  following  test  input  that  includes  a  grammatical  error: 

“The  very  new  working  Prime  was  kidnapped  Minister  by  terrorists.” 

Example:  In  Section  III  Noun  Phrases,  Subsection  5. 1.2. 3  Objective  Genitive,  one  of  the 
evaluators  used  the  following  test  input  that  includes  a  spelling  error:  the  word  “endorced” 
should  have  been  spelled  “endorsed.” 

“ARENA  endorced  the  terrorist’s  defiance.” 


9.  Inability  to  compose  test  input 

In  this  case,  the  evaluator  is  unable  to  compose  a  test  input  that  meets  the  specifications 
for  testing  the  linguistic  capability  identified  in  the  particular  Tool  item  and  assigns  a  score 
of  N. 

Example:  When  testing  comparative  adverbials,  one  evaluator  could  not  think  of  any 
adverbs  to  use  for  the  database  query  front-end  and  assigned  an  N  score,  but  other  evaluators 
came  up  with  and  used  adverbs  such  as  “earlier,”  “later,”  and  “more  recently”  that  the 
system  could  process. 
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Example:  In  Section  III,  Subsection  4.5.11  on  the  “PURPOSE-IS  Nominal  Compound 
Type,”  the  evaluators  had  trouble  thinking  up  test  inputs.  This  type  of  nominal  compound 
is  defined  to  be  a  noun-noun  phrase  in  which  the  first  noun  expresses  the  purpose  of  the 
the  entity  or  object  expressed  by  the  first  noun.  Examples  include  “delivery  truck”  and 
“bombing  raid.”  One  of  the  evaluators  used  these  two  examples  given  in  the  Tool  and  then 
stated  that  he  “Could  not  think  of  a  third  example”  and  assigned  the  system  a  score  of  N 
to  complete  his  three  tests  of  the  Tool  item. 


8.6  Flawed  Evaluation  Criteria  or  Procedure 

Another  factor  contributing  to  the  lack  of  agreement  across  interface  technologists  in  the 
consistency  study  seemed  to  be  that  the  procedure  or  evaluation  criteria  in  some  of  the 
individual  test  items  in  the  Tool  are  either  under-defined  or  flawed. 

Example:  In  Section  II  on  Interrogatives,  when  testing  the  text  processing  system,  the 
manner  of  testing  consisted  of  evaluators  entering  a  declarative  statement  to  inform  the 
system  of  some  fact(s)  and  then  entering  a  query  to  determine  whether  the  system  retained 
knowledge  of  the  fact(s).  When  the  facts  in  declarative  form  were  entered,  the  system  built 
a  template(s)  to  represent  the  information.  When  queried,  the  system  simply  presented  the 
template(s)  it  had  just  built. 

More  specifically,  Subsection  1.1  covers  the  use  of  “what”  as  an  interrogative  pronoun.  The 
criteria  for  the  Tool  item  is  stated  as: 


Criteria:  Provided  the  information  requested. 


The  issue  is  that  the  system  may  present  the  requested  information,  but  only  since  it  was 
just  entered.  Thus  any  correctness  in  its  output  is  an  artifact  of  the  manner  in  which  the 
test  is  being  conducted. 

Example:  In  Section  VI  on  Quantifiers,  Subsection  1.1. 2. 2  covers  Failed  Presuppositions 
and  the  criteria  is  stated  as: 


Criteria:  Demonstrated  understanding  that  an  error  occurred  because  the  amount  expressed 
by  the  postdeterminer  quantifier  following  the  definite  article  “the”  differed  from  the  known 
amount  of  the  entity  expressed  by  the  noun  phrase  being  quantified. 


When  given  a  sentence  in  which  the  presupposition  fails,  that  is,  a  sentence  where  the 
quantity  mentioned  is  incorrect  with  respect  to  the  context  (e.g.,  if  one  refers  to  “the  three 
managers”  then  it  is  presupposed  that  there  are  three  managers)  the  MUC-3  text  processing 
system  seems  to  understands  that  there  is  an  incongruity  between  the  quantities  of  the  entity 
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being  talked  about  and  any  entity  already  within  its  current  knowledge.  Instead  of  expressing 
that  there  is  an  error,  however,  the  system  assumes  that  the  input  refers  to  some  new  entity 
rather  than  referring  incorrectly  to  some  known  entity.  This  is  a  satisfactory  response  as 
noted  by  one  of  the  evaluators,  yet  our  criteria  did  not  account  for  this  possibility.  As  we  see 
below,  when  there  is  no  incongruity  a  final  template  is  created  which  reflects  the  specified 
number  of  the  entities.  When  there  is  an  incongruity  the  final  template  reflects  the  system’s 
understanding  that  there  is  more  than  one  group  of  entities  and  thus  the  number  of  targets 
is  additive. 

>  (nit  "Terrorists  bombed  two  banks  in  Anduhua.  The  two  banks  were  destroyed.") 


NIL 

0. 

MESSAGE  ID 

DEV-MUC3-0000 

1. 

TEMPLATE  ID 

1 

2. 

DATE  OF  INCIDENT 

-  31  OCT  89 

3. 

TYPE  OF  INCIDENT 

BOMBING 

4. 

CATEGORY  OF  INCIDENT 

TERRORIST  ACT 

5. 

PERPETRATOR:  ID  OF  INDIV(S) 

"Terrorists" 

6. 

PERPETRATOR:  ID  OF  ORG(S) 

- 

7. 

PERPETRATOR:  CONFIDENCE 

- 

8. 

PHYSICAL  TARGET:  ID(S) 

"banks" 

9. 

PHYSICAL  TARGET:  TOTAL  NUM 

2 

10.  PHYSICAL  TARGET:  TYPE(S)  OTHER:  "banks" 

11.  HUMAN  TARGET:  ID(S) 

12.  HUMAN  TARGET:  TOTAL  NUM 

13.  HUMAN  TARGET:  TYPE(S) 

14.  TARGET:  FOREIGN  NATION(S) 

15.  INSTRUMENT:  TYPE(S)  * 

16.  LOCATION  OF  INCIDENT  EL  SALVADOR 

17.  EFFECT  ON  PHYSICAL  TARGET(S)  DESTROYED:  "banks" 

18.  EFFECT  ON  HUMAN  TARGET (S) 

>  (nit  "Terrorists  bombed  three  banks  in  Anduhua.  The  two  banks  were  destroyed.") 


0. 

MESSAGE  ID 

DEV-MUC3-0000 

1. 

TEMPLATE  ID 

1 

2. 

DATE  OF  INCIDENT 

-  31  OCT  89 

3. 

TYPE  OF  INCIDENT 

BOMBING 

4. 

CATEGORY  OF  INCIDENT 

TERRORIST  ACT 

5. 

PERPETRATOR:  ID  OF  INDIV(S) 

"Terrorists" 

6. 

PERPETRATOR:  ID  OF  ORG(S) 

- 
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7. 

PERPETRATOR:  CONFIDENCE 

- 

8. 

PHYSICAL  TARGET:  ID(S) 

"banks 

"banks 

9. 

PHYSICAL  TARGET:  TOTAL  NUM 

5 

10. 

PHYSICAL  TARGET:  TYPE(S) 

OTHER: 

OTHER: 

11. 

HUMAN  TARGET:  ID(S) 

- 

12. 

HUMAN  TARGET:  TOTAL  NUM 

- 

13.  HUMAN  TARGET:  TYPE(S) 

14.  TARGET:  FOREIGN  NATION(S) 

15.  INSTRUMENT:  TYPE(S) 

16.  LOCATION  OF  INCIDENT 

17.  EFFECT  ON  PHYSICAL  TARGET(S) 

18.  EFFECT  ON  HUMAN  TARGET (S) 


EL  SALVADOR 
DESTROYED:  "banks" 


8.7  Complicating  Factors  from  NLP  System  Behavior 

We  observed  a  couple  factors  with  respect  to  the  behavior  of  the  NLP  systems  that  compli¬ 
cated  the  scoring  process  and  contributed  to  the  lack  of  agreement  across  Interface  Technol¬ 
ogists.  These  particular  factors  did  not  occur  frequently,  however.  The  following  two  items 
describe  the  types  of  system  behavior  and  provide  examples. 


1.  Conditional  system  behavior 

One  type  of  system  behavior  that  made  the  evaluation  process  more  difficult  was  system 
behavior  that  was  variable,  depending  on  the  existing  conditions,  when  the  evaluator  entered 
different  inputs  to  test  the  system’s  ability  to  handle  the  same  linguistic  phenomenon. 

For  example,  in  Section  VI  on  Quantifiers,  for  inputs  testing  a  failed  presupposition  combined 
with  a  cardinal  number  quantifier  (e.g.,  “the  two  women  in  the  NY  branch”),  the  database 
query  system  ignores  the  failed  presupposition  and  treats  the  phrase  as  a  query  about  any  of 
the  women  in  the  branch  when  the  number  mentioned  is  less  than  the  actual  number  in  the 
database,  but  acknowledges  the  failed  presupposition  when  the  number  mentioned  exceeds 
the  actual  number.  Thus,  the  evaluator  needs  to  test  the  different  possible  cases  in  order  to 
avoid  an  inaccurate  score  assignment  for  the  system.  The  evaluation  Tool  could  be  improved 
to  account  for  such  system  behavior. 


2.  Ignoring  Test  Input 

Another  type  of  system  behavior  that  made  the  evaluation  process  more  complicated  was  the 
phenomenon  of  the  system  ignoring  the  critical  part  of  the  test  input.  That  is,  a  complicating 
factor  was  the  fact  that  an  NLP  system  would  occasionally  ignore  a  critical  part  of  the  NL 
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input,  and  process  and  respond  to  a  modified  input. 

For  example,  when  testing  the  ability  of  the  database  query  front-end  system  to  understand 
the  word  “neither,”  the  following  test  input  was  used.  Note  that  there  are  two  people  with  a 
last  name  of  “Smith”  in  the  employee  database.  When  the  sentence  is  input  to  the  system, 
it  ignores  the  critical  word  “neither”  and  responds  with  information  about  each  department 
that  does  include  a  Smith. 

“List  the  department  that  includes  neither  Smith.” 
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8.8  Subjective  Assessment  of  the  NLP  Evaluation  Tool 

Following  the  application  of  the  Evaluation  Tool  each  Technologist  was  asked  to  complete 
a  questionnaire  designed  to  assess  how  well  the  Technologist  felt  the  Tool  worked.  Informal 
interviews  were  held  following  completion  of  the  questionnaires  to  clarify  and  expand  on  the 
responses,  as  needed.  This  section  contains  a  high-level  summary  of  information  obtained 
from  the  questionnaires.  A  more  detail  presentation  of  the  information  from  the  question¬ 
naires  and  informal  interviews  is  included  in  Appendix  B.  In  some  areas  there  was  significant 
agreement  across  subjects  while  in  other  areas  opinions  differed. 

The  results  of  the  questionnaire  and  subsequent  interviews  brought  out  many  issues 
and  provided  insights  into  the  strengths  and  weaknesses  of  the  evaluation  tool.  Some  of  the 
more  relevant  points  are  shown  below: 

•  Technologists  all  felt  that  the  scoring  criteria  need  to  be  made  more  explicit.  They 
called  for  criteria  that  could  applied  without  special  training  or  ill-defined  judgements. 
For  example,  there  was  some  objection  to  the  term  “demonstrated  understanding.” 
The  technologists  also  wanted  more  explicit  instructions  about  the  standard  of  com¬ 
parison.  They  asked  whether  understanding  was  to  be  demonstrated  to  a  level  that 
would  be  expected  of  a  human,  or  whether  standards  could  be  relaxed  to  overlook 
small  shortcomings  since  a  machine  was  being  evaluated. 

•  The  examples  illustrating  each  test  item  were  found  to  be  valuable  for  understanding 
what  was  being  evaluated.  In  addition,  the  examples  were  used  by  the  technologists  to 
form  NLP  inputs  directly.  It  was  possible  to  use  the  examples  in  this  way  because  they 
were  directly  relevant  to  the  NLPs  that  were  evaluated.  This  will  not  always  be  the 
case  and  there  is  a  danger  of  inconsistent  results  across  NLPs  due  to  varying  degrees 
of  relevance  of  the  examples.  One  technologist  felt  that  the  direct  relevance  and  use 
of  examples  limited  the  range  of  inputs  that  were  tested  and  that  examples  should  be 
more  generic. 

•  The  technologists  felt  that  the  tool  did  not  provide  equally  valid  results  for  both  NLP 
systems  that  were  evaluated.  Some  felt  that  the  evaluation  of  the  database  NL  query 
system  provided  more  valid  results  than  the  evaluation  of  the  MUC-3  text  processing 
system.  Part  of  the  problem  is  that  technologists  believed  the  text  processing  system 
was  not  understanding  NL  inputs  even  though  outputs  were  essentially  correct.  Part 
of  the  problem  was  that  the  text  processing  system  provided  outputs  in  the  form  of  a 
template.  It  was  not  always  possible  to  adequately  assess  “understanding”  from  the 
template. 

•  All  technologists  felt  that  the  training  they  received  was  necessary  for  the  proper  use  of 
the  evaluation  Tool.  This  seemed  to  apply  primarily  to  the  application  of  the  criteria 
and  score  assignment  aspects  of  the  evaluation  process  rather  that  the  mechanics  and 
rules  for  using  the  Tool.  Some  expressed  the  feeling  that  more  explicit  criteria  could 
reduce  the  need  for  training. 
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•  A  few  technologists  stated  that  they  needed  to  go  back  and  review  test  items  that  were 
evaluated  early  in  the  Tool  because  of  new  insights  gained  while  evaluating  later  items. 
This  problem  (if  it  is  a  problem)  may  be  overcome  or  reduced  with  more  extensive 
training.  Some  technologists  expressed  their  feeling  that  the  tool  should  be  applied  in 
two  passes  to  take  advantage  of  this  learning  effect  and  thereby  increase  consistency 
of  scoring. 

•  Some  of  the  technologists  suggested  that  the  scores  be  refined  to  allow  more  fine-grained 
identification  of  NLP  shortcomings  (e.g.,  wouldn’t  parse  input  vs.  produced  garble  as 
output). 

•  The  greatest  strength  according  to  most  of  the  technologists  is  that  it  is  very  com¬ 
prehensive.  This  strength  should  be  understood  in  the  context  of  what  was  generally 
felt  to  be  the  Tool’s  greatest  weakness,  that  it  is  very  tedious  and  time-consuming  to 
apply.  One  technologist  described  it  as  a  “brain-dead  work.”  Two  of  the  technologists 
suggested  limiting  evaluation  sessions  to  two  to  four  hours  and  expressed  concern  that 
boredom  would  affect  scoring. 

•  One  technologist  stated  that  the  exhaustiveness  of  the  tool  is  useful  in  evaluating  NLP 
systems  because  it  forces  evaluators  to  experience  all  NL  structures.  This  technologist 
notes  that  after  applying  the  tool  you  have  a  pretty  good  sense  of  the  usefulness  of 
the  NLP  for  the  considered  application  irrespective  of  the  actual  scores  produced.  The 
experience  itself  is  valuable. 

•  One  technologist  expressed  the  idea  that  the  tool  is  valuable  not  only  for  evaluating 
existing  NLPs,  but  is  also  potentially  useful  as  a  tool  for  guiding  the  development  of 
NLPs. 
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9.  Conclusions  and  Recommendations 
9.1  Conclusions 

The  NLP  Evaluation  Tool  fulfills  the  program  objectives: 

•  Produces  profiles  of  NLP  system  capabilities: 

—  Descriptive,  Quantitative,  Hierarchical 

•  Applicable  across  application  domains 

•  Applicable  to  different  types  of  NLP  systems 

•  Does  not  require  the  evaluator  to  be  a  trained  linguist 

•  Repeatable,  with  consistent  results  across  evaluators  (consistency  partially  fulfilled) 

•  Unbiased  with  respect  to  linguistic  theory 

•  Based  on  current  and  future  needs  and  capabilities 

•  Based  on  human-human  NL  communication  as  a  model 

The  NLP  Evaluation  Tool  provides: 

•  A  test-construction  facility 

•  A  test  application  facility 

•  A  profiling  facility 

•  Analysis  capability 

The  Interface  Technologists  who  participated  in  the  final  assessment  exercise  seemed  to 
feel  that  the  greatest  strength  of  the  Tool  is  that  it  is  very  comprehensive.  When  you 
get  through,  there  is  a  sense  that  the  NLP  system  has  been  exhaustively  tested  in  a  very 
structured  manner.  Additional  positive  comments  by  the  technologists  included: 

•  The  exhaustiveness  of  the  tool  is  useful  in  evaluating  NLP  systems  because  it  forces 
evaluators  to  experience  all  NL  structures. 

•  After  applying  the  tool  you  have  a  pretty  good  sense  of  the  usefulness  of  the  NLP  for 
the  considered  application  even  without  the  actual  scores  produced. 

•  The  Tool  is  potentially  useful  as  a  tool  for  guiding  the  development  of  NLPs. 
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The  Tool  is  not  without  its  drawbacks,  however.  Primarily,  the  shortcomings  are  that  (1) 
the  Tool  is  very  tiring  and  time-consuming  to  apply  and  that  (2)  the  Interface  Technologists 
applying  the  Tool  as  part  of  the  reliability  study  at  the  end  of  the  program  did  not  produce 
consistent  system  profiles  in  all  sections  of  the  Tool.  These  issues  were  discussed  at  length 
in  Section  8. 

As  a  result  of  the  shortcomings  of  the  Tool,  we  have  identified  a  set  of  tasks  that  we  recom¬ 
mend  to  improve  the  Tool  in  the  areas  listed  below.  These  recommendations  are  discussed 
in  more  detail  in  the  following  subsection. 

•  Improvement  of  the  Tool  application  process 

•  Improvement,  refinement,  and  completion  of  the  Tool 

•  Development  of  a  computer-based  version  of  the  Tool 

•  Improvement  and  standardization  of  evaluator  training 

•  Development  of  a  Qualifying  Test 

•  Performance  of  a  follow-on  assessment  exercise 


9.2  Recommendations 

Recommendation  #  1.  Tool  Application  Process 

Training  and  Testing: 

The  evaluators  (interface  technologists)  should  definitely  undergo  training  on  the  Tool  and 
the  NLP  systems  and  their  application  domains.  This  was  done  in  each  of  our  assessment  ex¬ 
ercises  and  was  viewed  as  essential  by  the  participating  interface  technologists.  We  feel  that 
the  training  should  be  made  more  standardized  and  should  be  supplemented  by  a  standard¬ 
ized  qualifying  test.  Standardized  training  and  testing  will  ensure  consistent  results/profiles 
from  the  Tool.  This  topic  is  discussed  in  more  detail  as  part  of  Recommendation  #2. 

Score  Choices: 

Retain  the  scoring  choices  and  their  definitions:  S,  C,  P,  F,  N.  The  feedback  from  the 
interface  technologists  that  participated  in  the  final  assessment  exercise  indicates  that  the 
score  choices  are  adequate. 
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Number  of  tests  per  Tool  item: 

It  should  be  made  a  rule  for  applying  the  NLP  Evaluation  Tool  that  each  evaluator  enters  as 
many  test  inputs  as  necessary,  for  each  evaluation  item,  to  feel  confident  that  the  score  being 
assigned  by  the  evaluator  is  truly  representative  of  the  capability  of  the  NLP  system  on  the 
particular  linguistic  phenomenon  being  tested.  We  further  recommend  that  the  evaluator 
enter  just  one  score  for  each  evaluation  item  and  that  the  evaluator  may  split  the  score 
among  the  possible  score  choices  (e.g.,  75%  S,  25%  C),  if  the  evaluator  feels  that  this  is 
representative  of  the  system’s  behavior  on  the  given  linguistic  phenomenon. 

During  the  last  exercise  to  assess  the  Evaluation  Tool,  the  evaluators  entered  three  test  inputs 
for  each  Tool  evaluation  item  and  entered  a  score  for  the  system’s  processing  of  each  test 
input,  but  there  was  no  effort  on  the  part  of  the  evaluators  to  be  sure  that  the  scores  entered 
were  truly  representative  of  the  system’s  behavior.  For  example,  sometimes  an  evaluator 
would  “count”  a  test  input  that  used  a  word  or  object  that  is  not  within  the  domain  of  the 
system  (e.g.,  asking  the  database  query  system  about  the  residences  of  employees).  This  type 
of  input  should  not  be  counted  against  the  system,  but  should  be  disgarded  and  ignored. 

Splitting  Scores: 

Since  we  recommend  that  the  evaluator  be  given  the  responsibility  for  determining  a  rep¬ 
resentative  score  for  each  Tool  item,  we  feel  that  the  evaluator  should  be  allowed  to  split 
his/her  score  among  the  poss;ble  score  choices  (e.g.,  75%  S,  25%  C).  From  our  experience, 
It  seems  that  it  is  not  always  possible  for  the  evaluator  to  determine  a  score  that  is  repre¬ 
sentative  across  all  the  variations  in  test  inputs  allowable  according  to  the  specification  of  a 
Tool  item  (see  Section  8.6)  and  therefore  a  split  would  be  appropriate  in  such  cases. 

Recommendation  #  2.  Improvement /Refinement  of  the  Tool 

The  contents  of  the  Tool  should  be  improved  and  revised  using: 

•  information  gathered  from  the  Phase  3  Tool  assessment  exercise  such  as: 

—  agreement/disagreement  data  and  analysis  results, 

—  comments  made  by  the  interface  technologists  on  individual  items  during  their 
application  of  the  Tool, 

—  feedback  from  the  interface  technologists  in  response  to  the  Assessment  Question¬ 
naire 

•  input  from  two  experienced  computational  linguists  after  their  review  of  the  Tool. 
Selected  recommended  modifications  include: 

•  Make  some  of  the  evaluation  criteria  more  specific,  where  necessary, 
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•  Provide  more  detailed  instructions  on  how  to  test  certain  of  the  more  difficult  items, 


•  Provide  more  definitive  explanations  of  what  is  acceptable  output  for  some  of  the  more 
difficult  items, 

•  Provide  example  output  for  some  items,  especially  in  some  of  the  early  sections  and 
for  the  more  difficult  items  throughout  the  Tool, 

•  Provide  scoring  guidance  for  items,  where  needed. 


Recommendation  #  3.  Computer-based  Tool 

Develop  a  computer-based  version  of  the  NLP  Evaluation  Tool  tool  that  provides  the  evai- 
uator  with  automated  support  in  areas  such  as: 

•  Access  to  the  NLP  system’s  lexicon  by  lexical  category 

•  Access  to  examples  of  different  types  of  words/phrases/clauses,  indexed  by  category 

•  Editing,  retrieving,  assembling,  and  storing  test  text/sentences 

•  Spelling  checker 

•  Grammar  checker 

•  Indexed  storage  and  retrieval  of  the  types  of  linguistic  phenomena  on  which  the  NLP 
system  has  succeeded  and  failed 

•  Storage  and  retrieval  of  actual  test  inputs  indexed  by  system  performance  and  by 
linguistic  category 

•  Automatic  submission  of  test  input  to  the  NLP  system 

•  Recording  the  session  with  the  NLP  system 

•  Automatic  aggregation  of  scores  for  each  linguistic  phenomenon  class 

•  Automatic  generation  of  NLP  system  profiles  at  different  levels  of  granularity  of  the 
linguistic  classification  scheme 

•  Automatic  generation  of  NLP  syctem  profiles  showing  different  emphasis  or  perspective 
within  the  classification  scheme 
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Recommendation  #  4.  Training/Tutorial 

A  short  formal  self-instructional  tutorial  (mini-course)  should  be  designed  to  train  interface 
technologists  in  the  use  of  the  NLP  Evaluation  Tool.  The  tutorial  should  culminate  in  a 
qualifying  test  that  the  interface  technologist  should  pass  before  actually  applying  the  Tool 
to  any  NLP  system.  This  tutorial  should  be  designed  to  cover  all  the  important  features  and 
issues  associated  with  using  the  Tool  and  all  problem  areas  that  have  been  identified.  At  a 
high  level,  coverage  should  include: 

•  The  purpose  of  the  NLP  Evaluation  Tool 

•  Organization  and  use  of  the  Evaluation  Tool 

•  The  procedure  for  applying  the  Tool 

•  How  to  use  the  computer-based  facilities 

•  The  NLP  systems 

At  a  more  detailed  level,  the  Tutorial  should  cover  the  important  issues  associated  with  using 
the  Tool: 

•  The  Tool  application  principle  of  using  only  linguistic  phenomena  on  which  the  NLP 
system  has  succeeded,  except  for  the  particular  phenomenon  being  tested 

•  Constructing  NL  inputs  that  test  the  particular  linguistic  phenomenon 

•  Using  variety  in  test  inputs 

•  Application  of  evaluation  criteria 

•  Judging  acceptable  output 

•  Scoring  a  system’s  output 

Important  design  features  for  the  Tutorial  are: 

•  The  tutorial  should  be  computer-based,  if  possible 

•  Include  illustrative  examples  with  explanatory  material 

•  Include  exercises  that  require  the  interface  technologist  to  work  through  selected  pre¬ 
scored  items  from  the  Evaluation  Tool 

•  Include  exercises  to  test  understanding  and  provide  immediate  feedback  (e.g.,  multiple 
choice  questions) 
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•  Include  questions  that  require  the  interface  technologist  to  find  the  errors  in  negative 
examples  of  Tool  application 

•  Include  “hands-on”  application  of  selected  Tool  sections  to  at  least  one  NLP  system 

Recommendation  #  5.  Qualifying  Test 

Develop  a  qualifying  test  that  is  designed  to  follow  after  the  tutorial.  A  standard  qualifying 
test  would  help  ensure  consistency  of  the  Tool  results  (i.e.,  NLP  system  profiles)  across 
evaluators. 

Desireable  features  for  the  Test  include: 

•  Coverage  should  include  all  the  important  Tool  application  features  and  issues  covered 
in  the  tutorial 

•  The  test  should  be  computer-based,  if  possible 

•  Use  question  types  that  provide  immediate  feedback  (e.g.,  multiple  choice) 

Recommendation  #  6.  Follow-On  Assessment  Exercise 

We  recommend  that  another  assessment  exercise  be  conducted  after  accomplishment  of  one 
or  more  of  the  above  listed  recommended  actions.  The  objective  of  the  exercise  would  be  to 
determine  the  reliability  of  the  Tool  after  having  developed  a  tutorial  and  improved  training, 
revised  the  evaluation  Tool,  developed  computer-based  facilities  to  assist  the  evaluator  in 
applying  the  Tool,  and  improved  the  procedure  for  applying  the  Tool. 

We  recommend  that  the  exercise  be  conducted  in  a  manner  that  is  similar  to  the  last  Phase 
3  exercise,  but  with  some  changes.  The  following  summarizes  the  key  features  of  the  recom¬ 
mended  exercise: 

•  NLP  systems:  At  least  two  (2)  mature  systems 

•  Interface  Technologists: 

-  At  least  four  (4)  people 

—  Separate  and  distinct  from  Tool  development  and  review  people 

-  Competent  users  of  English,  but  not  trained  linguists 

•  Perform  thorough  training  of  the  interface  technologists 

-  Tutorial 

-  Qualifying  test 
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-  Training  on  NLP  application  domain 

•  Use  a  Latin  Square  design  to  avoid  bias  due  to  order 

•  For  each  NLP  system,  have  one  of  the  developers  review  the  evaluation  results 

•  Perform  consistency  analysis 
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Introduction 


1.  Objective 

The  NLP  Evaluation  Tool  is  designed  to  test  NLP  systems  to  determine  which  linguistic 
capabilities  they  possess  and  which  they  do  not  possess.  This  Tool  is  designed  to  produce 
a  profile  of  an  NLP  system  identifying  the  lexical,  syntactic,  semantic,  and  discourse  phe¬ 
nomena  that  the  system  can  handle  and  those  that  it  cannot.  Although  this  Tool  provides 
detailed  coverage  of  a  wide  variety  of  linguistic  phenomena,  we  make  no  claim  of  complete¬ 
ness.  A  few  subsections  are  marked  “t.b.d.”  to  indicate  areas  that  we  have  identified  to  be 
developed. 

2.  Design  Principle 

The  Tool  is  designed  to  progress  from  elementary  sentence  types  containing  simple  con¬ 
stituents  to  more  complex  sentence  types.  The  principle  is  that  each  time  test  sentences  are 
presented  to  the  NLP  system  being  evaluated,  the  sentences  should  contain  only  one  new 
(untested)  linguistic  phenomenon  or  one  new  untested  combination  of  tested  phenomena. 
The  other  capabilities  that  are  required  for  processing  the  input  should  already  have  been 
tested  and  the  NLP  system  should  already  have  succeeded  on  these  other  phenomena.  If 
the  NLP  system  clearly  fails  on  a  certain  type  of  linguistic  phenomenon  or  feature,  then  the 
evaluator  should  not  include  that  phenomenon  or  feature  again  in  subsequent  test  sentences. 

In  this  way,  each  Tool  test  item  tests  just  one  untested  NLP  capability  at  a  time,  to  the 
extent  possible,  and  combinations  are  tested  after  the  individual  capabilities  are  tested.  The 
intent  is  to  make  the  Tool  sensitive  to  each  individual  linguistic  capability. 

3.  Evaluation  Tool  Features 

The  Evaluation  Tool  is  designed  to  be  domain  independent.  It  therefore  does  not  include  nor 
rely  on  a  predefined  set  of  natural  language  texts  or  sentences.  Instead,  the  test  sentences  or 
paragraphs  to  be  processed  by  the  NLP  system  are  composed  or  provided  by  the  evaluator 
either  during,  or  prior  to,  the  administration  of  the  Tool.  The  Tool  is  designed  to  assist  the 
evaluator  with  the  creation,  modification,  or  tailoring  of  test  sentences. 

Since  the  Tool  is  designed  for  use  by  people  who  are  not  well  versed  in  linguistics,  the  Tool 
includes  explanatory  material  that  is  intended  to  provide  sufficient  instruction  to  enable  the 
evaluator  to  compose  proper  test  sentences.  A  glossary  is  provided  to  aid  the  evaluator. 

4.  Structure 

Each  Procedure  item  consists  of  the  following  components: 

•  A  brief  explanation  and  definition  of  the  linguistic  capability  being  tested,  along  with 
any  special  instructions  for  testing.  This  is  particularly  important  for  evaluators  that 
do  not  have  any  linguistic  background. 
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•  Patterns  that  define  the  structure  and  features  of  the  test  sentences  to  be  composed 
and  submitted  to  the  NLP  system  under  evaluation.  The  specific  words  of  the  test 
sentences  are  supplied  by  the  evaluator,  appropriate  for  the  particular  application  for 
which  the  NLP  system  has  been  installed  and  with  which  it  executes. 

•  Example  sentences  to  aid  the  evaluator  in  composing  test  sentences. 

•  Criteria  against  which  to  evaluate  the  NLP  system’s  behavior. 

•  A  place  for  the  evaluator’s  test  sentences. 

•  A  place  for  the  evaluator’s  score. 


5.  Scoring 

For  each  test  item  in  the  Tool,  the  evaluator  submits  natural  language  test  inputs  to  the 
NLP  system  being  evaluated  and  determines  whether  or  not  the  responses  indicate  that  the 
system  understood  and  processed  the  inputs  correctly.  The  evaluator  has  five  choices  of 
scores  to  award  to  the  system  for  each  test  item  as  listed  below. 

•  Success  (S):  The  system  successfully  met  the  evaluation  criteria  stated  for  the  partic¬ 
ular  test  item. 

•  Correct  (C):  The  system  did  not  successfully  meet  the  evaluation  criteria,  but  produced 
acceptable/correct  output. 

•  Partially  Correct  (P):  The  system  did  not  successfully  meet  the  evaluation  criteria, 
and  only  produced  partially  acceptable/correct  output. 

•  Failure  (F):  The  system  did  not  successfully  meet  the  evaluation  criteria  and  produced 
no  correct  output. 

•  No  Output  (N):  The  system  produced  no  output. 
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Guide  to  Notation 


Abbreviation 

Meaning 

Page 

[Adj] 

Adjective 

1-5, VIM 

[Adj.Nom] 

Adjective  Nominal 

III-2 

[Adjective] 

Adjective 

1-5,111-8 

[Adjective  -{-compare] 

Comparative  Adjective 

VII-1 

[Adjective  -Fsuper] 

Superlative  Adjective 

VII-18 

[Adverb] 

Adverb 

IV-1 

[Adverb  +compare] 

Comparative  Adverb 

VII-20 

[Adverbial] 

Adverbial 

IV-4 

[AdvP] 

Adverb  Phrase 

IV-1 

[Anaphoric  NP] 

Anaphoric  Noun  Phrase 

X-8,X-1 

[Antecedent  NP] 

Antecedent  Noun  Phrase 

X-3 

[Antecedent  S] 

Antecedent  Sentence 

X-7 

[Antecedent  VP] 

Antecedent  Verb  Phrase 

X-6 

[Aux] 

Auxiliary  Verb 

1-4 

[Aux-Verb] 

Auxiliary  Verb 

X-ll 

[BE-Verb] 

Form  of  the  verb  “to  be” 

MO 

[Central] 

Central 

III-7 

[Central  Premodifier] 

Central  Premodifier 

III-7 

[Complement] 

Complement 

1-3 

[Conj  -f  subord] 

Subordinating  Conjuction 

X-3 

[Copular] 

Copular  Verb 

1-3 

[Count  Noun] 

Count  Noun 

1-1,1- 7 

[Def  Article] 

Definite  Article 

1-7 

[Det] 

Determiner 

1-8,111-7 

[Determinative] 

Determinative  element 

111-1,111-4,111-7 

[Determiner] 

Determiner 

I- 1,1-8 

[DO- Verb] 

Form  of  the  verb  “do” 

1-4 

[Have- Verb] 

Form  of  the  verb  “have” 

V-17 

[Head] 

Head  of  a  phrase 

III-l 

[Indef  Article] 

Indefinite  Article 

1-6 

[Indef  Quant  Pronoun] 

Indefinite  Quantifier  Pronoun 

VI-21 

[Mass  Noun] 

Mass  Noun 

1-9 

[Noun] 

Noun 

1-1 

[NP] 

Noun  Phrase 

1-1 

[NP  -{-conjunction] 

Noun  Phrase  with  conjunction 

VIII-4 

[NP  -det] 

Noun  Phrase  minus  determinative 

III-4,VI-1 

Guide  to  Notation  (Continued) 


Abbreviation 

Meaning 

Page 

[NP  +DET  QUANT] 

Noun  Phrase  with  determiner  quantifier  VI-2 

[NP  +OEN1TIVE] 

Noun  Phrase  in  genitive  case 

111-20,111-24 

[NP  +PRo] 

Pronoun 

X-2 

[Object] 

Object 

I-12,VII-31,XI-9 

[Participle] 

Participle 

III-3,V-8 

[Participle  (-ed)] 

Passive  Participle 

1-2,111-3 

[Participle  (-ing)] 

Progressive  Participle 

1-2,111-4 

[Postcedent  NP] 

Postcedent  Noun  Phrase 

X-14 

[PP] 

Prepositional  Phrase 

1-8,111-1 

[Prep] 

Preposition 

III-l,X-9 

[Precentral] 

Precentral 

III-7 

[Postmodification] 

Postmodification  of  a  Noun  Phrase 

III-l 

[Preposition] 

Preposition 

11-5,111-1 

[Proper  Noun] 

Proper  Noun 

1-9 

[Premodification] 

Premodification  of  a  Noun  Phrase 

III-l 

[Rel  Pronoun] 

Relative  Pronoun 

III-32 

[Relative  Clause] 

Relative  Clause 

III-32 

[SubjCompl] 

Subject  Complement 

1-3,1-10 

[Verb] 

Verb 

I-10,V-1 

[Verb  (-ed)] 

-ed  participle 

1-2, V-8 

[Verb  (-ing)] 

-ing  participle 

I-2,V-12 

[Verb  -4- active] 

active  verb 

1-15 

[Verb  -f ditrans] 

Ditransitive  Verb 

1-14 

[Verb  -4-inf] 

Infinitive  Verb 

III-45 

[Verb  +intrans] 

Intransitive  Verb 

1-12 

[Verb  -(-passive] 

-ed  participle  of  verb 

I- 15,  V-8 

[Verb  -(-pres] 

Present  tense  of  Verb 

V-6 

[Verb  -(-past] 

Past  tense  of  Verb 

V-7 

[Verb  -(-future] 

Future  tense  of  Verb 

V-8 

[Verb  +proo] 

Progressive  Verb 

III-38 

[Verb  -f trans] 

Transitive  Verb 

1-12 

[Verb  Group] 

Verb  Group 

[Verb  Head] 

Verb  Head 

XI-7 

[VG] 

Verb  Group 

[VP] 

Verb  Phrase 

1-3, V-l 

[Wh-Word] 

Word  beginning  with  Wh 
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THE  EVALUATION  PROCEDURE 


I.  BASIC  SENTENCES 

Noun  phrases  and  verb  phrases  axe  the  basic  components  of  English  sentences.  We  will 
describe  these  components  before  testing  the  three  basic  sentence  types:  declarative ,  imper¬ 
ative,  and  interrogative.  First  we  will  discuss  determiners  and  nouns,  which  are  the  basic 
parts  of  noun  phrases. 

Simple  Determiners 

The  only  determiners  we  will  use  in  this  section  are  the  articles.  These  are: 

•  Definite  article:  the 

•  Indefinite  article:  a,  an 

Nouns 

•  Count  Nouns:  Nouns  which  denote  individual  countable  entities,  such  as 

—  Singular:  Chairman,  employee,  department,  salary,  mayor,  civilian,  terrorist, 
mouse,  etc. 

—  Plural:  Chairmen,  employees,  departments,  salaries,  mayors,  civilians,  terrorists, 
mice,  etc. 

•  Proper  Nouns:  Nouns  which  name  specific  persons,  places,  and  things  such  as  John 
Smith,  Peru,  Santa  Elena,  Andahua,  San  Salvador,  etc. 

•  Mass  Nouns:  Noun  which  denote  an  undifferentiated  mass  or  continuum  such  as  expe¬ 
rience,  money,  information,  work,  water,  bread,  grass,  warmth,  music,  sand,  overhead, 
etc. 


Simple  Noun  Phrases 

The  noun  phrase,  denoted  [NP],  is  an  important  component  of  English  sentences.  Simple 
noun  phrases  frequently  consist  of  a  noun  alone,  or  a  determiner  followed  by  a  noun. 

In  the  following  pattern-example  diagram,  we  include  two  patterns.  The  second  pattern 
(below  the  first)  is  an  elaboration  of  the  first.  Parentheses  are  used  to  enclose  optional 
elements  of  the  structure  or  pattern. 
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Pattern: 


Examples: 


[NP] 

([Determiner])  [Noun] 
an  employee 

salespersons 
John  Smith 

the  president 

information 
the  data 


Verbs 

The  purpose  of  this  section  is  to  introduce  some  of  the  basic  categories  and  attributes  of  verbs. 
The  information  introduced  here  is  extended  in  subsequent  sections  later  in  the  Procedure. 
This  section  and  the  subsequent  section  on  simple  verb  phrases  should  provide  sufficient 
information  about  verbs  and  verb  phrases  to  enable  you  to  compose  simple  sentences. 

There  are  three  major  categories  of  verbs: 

•  The  Full  Verbs  (or  lexical  verbs)  such  as  call,  leave,  walk,  explode,  etc.  Note:  This 
category  is  called  an  open  class  since  new  verbs  can  continually  be  added  to  this  class 
and  therefore  the  number  of  words  in  the  class  is  unlimited. 

•  The  Primary  Verbs  :  be,  have,  do.  Note:  This  category  is  called  a  closed  class  since 
new  words  are  rarely  added  to  this  class. 

•  The  Modal  Auxiliary  Verbs  :  will,  might,  could,  etc.  Note:  This  is  a  closed  class  also. 

The  full  verbs  can  only  act  as  main  verbs  in  a  verb  phrase,  the  modal  auxiliary  verbs  can 
act  only  as  auxiliary  verbs,  and  the  primary  verbs  can  act  as  either  main  or  auxiliary  verbs. 

Regular  full  verbs  have  four  morphological  forms  while  irregular  full  verbs  vary  in  this  respect. 
Some  of  these  morphological  forms  will  be  used  in  the  explanation  of  simple  verb  phrases. 
They  will  be  used  more  extensively  in  a  later  section  on  more  complex  verb  phrases. 


Verb  Forms 

Form 

Regular  Verbs 

Irregular  Verbs 

Base  Form 

walk 

leave 

build 

speak 

-s  Form 

walks 

leaves 

builds 

speaks 

-ing  Form 

walking 

leaving 

building 

speaking 

Past  Form 

walked 

left 

built 

spoke 

-ed  Participle 

walked 

left 

built 

spoken 
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Simple  Verb  Phrases 

The  purpose  of  this  section  is  to  provide  sufficient  discussion  of  simple  verb  phrases  to  enable 
you  to  compose  simple  sentences.  The  discussion  in  this  section  is  very  basic  and  is  extended 
in  later  sections  of  this  Procedure. 

A  simple  verb  phrase  consists  of  the  present  tense  form  of  a  full  verb  (call,  leave,  walk), 
perhaps  followed  by  a  noun  phrase. 

Below  are  some  simple  verb  phrases.  As  seen  above,  in  the  noun  phrase  examples,  certain 
pattern-example  diagrams  have  two  patterns.  The  second  pattern  (below  the  first)  is  an 
elaboration  of  the  first. 


Pattern: 

[VP] 

[Verb] 

Examples: 

walks 

leave 

Pattern: 

[VP] 

[Verb] 

[NP] 

Examples: 

hire 

employees 

kidnap 

the  mayor 

Examples  of  sentences  with  simple  verb  phrases: 

Pattern:  [NP]  [VP] 

Examples:  The  manager  hires  employees. 

John  Smith  works. 

Terrorists  kidnapped  the  mayor. 

A  simple  copular  verb  phrase  consists  of  the  present  tense  form  of  the  BE- verb  (am,  is, 
are )  followed  by  a  subject  complement,  that  is,  an  adjective  or  a  noun  phrase  which  provides 
additional  description  or  specification  of  the  subject  of  the  sentence.  A  simple  type  of  copular 
verb  phrase  consists  of: 

Pattern:  [BE- Verb]  [Subject  complement] 

[BE- Verb]  [NP]  or  [Adjective] 

Examples:  is  intelligent 

is  male 

are  managers 

Examples  involving  the  use  of  the  copular: 


Declarative  Sentences. 


Pattern: 

[NP] 

[VP] 

[NP] 

[BE-Verb] 

[Subject  Complement]. 

Examples: 

The  manager 

is 

a  woman. 

John  Smith 

is 

an  employee. 

Yes/No-Questions. 

President  Cerezo 

is 

unpopular. 
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Pattern:  [BE- Verb]  [NP]  [Subject  Complement]? 

Examples:  Is  John  Smith  an  employee? 

Is  John  Smith  male? 

The  verb  DO  can  be  used  both  as  an  auxiliary  and  as  a  main  verb.  The  main  uses  of  DO  as 

an  auxiliary  are: 

•  in  indicative  clauses  negated  by  not  as  in  “ she  did  not  complete  the  M.A.  program” 

•  in  questions  and  other  constructions  that  involve  subject-verb  inversion,  where  the 
verb  is  in  the  simple  present  or  past  tense,  as  in  “ Did  John  finish  the  report  ?” 

•  in  emphatic  constructions  where  the  verb  is  simple  present  or  simple  past  tense,  as  in 
“the  chairman  does  want  you  to  attend  the  meeting.” 

•  in  reduced  clauses  where  DO  acts  as  a  dummy  operator  preceding  ellipsis  of  a  predi¬ 
cation,  as  in  “John  received  a  bonus  and  so  did  Mary.” 

In  this  section  on  simple  verb  phrases,  we  will  cover  the  use  of  DO  in  interrogates  and 
emphatic  constructions.  The  other  cases  will  be  covered  and  used  in  later  sections  of  the 
Procedure. 

The  structure  of  a  verb  phrase  which  includes  the  auxiliary  verb  DO  is  generally  as  follows: 

Pattern:  [VP] 

[DO-Verb]  [Verb]  ([NP]) 

Examples:  did  work 

does  earn  $30,000 

did  join  Xidec 

Examples  involving  the  use  of  DO: 


Wh-Questions. 


Pattern: 

[Wh-Word] 

([NP]) 

[DO-Verb] 

[NP] 

[VP]  ? 

Examples: 

What 

does 

John 

earn? 

When 

did 

John 

join  Calspan? 

What 

department 

did 

John  Smith 

join? 

Yes/No-Questions. 

Pattern: 

[DO-Verb] 

[NP]  [VP]  ? 

Examples: 

Does 

John  have  a 

degree  ? 

Does 

John  earn  $30,000  ? 

Declarative  Sentences. 

Pattern: 

[NP] 

[DO-Verb] 

[VP] 

Examples: 

The  terrorists  did 

kidnap  the  mayor. 

The  ELN 

did 

bomb  the  bank. 
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1.  Basic  Sentence  Types 

1.1  Declarative  Sentences 

A  declarative  sentence  is  primarily  used  to  convey  information,  that  is,  to  state  propositions 
and  make  assertions.  Typically  a  declarative  sentence  consists  of  a  noun  phrase  (subject) 
followed  by  a  verb  phrase,  which  may  be  a  full  verb  or  a  full  verb  followed  by  a  [NP],  or  may 
be  a  copular  verb  phrase  (a  [BE- Verb]  followed  by  a  [NP]  or  [Adjective]). 


Pattern: 

[NP] 

[VP] 

[NP] 

[Verb] 

[NP]. 

Examples: 

John  Smith 

earns 

$30,000. 

Terrorists 

kidnapped 

the  mayor. 

Guerrillas 

attacked 

the  farm. 

Pattern: 

[NP] 

[VP] 

[NP] 

[BE-Verb] 

[Adjective]. 

Examples: 

John  Smith 

is 

married. 

Jane  Doe 

is 

female. 

The  assassin 

was 

clever. 

Pattern: 

[NP] 

[VP] 

[NP] 

[BE-Verb] 

[NP]. 

Examples: 

Jane  Doe 

is 

a  manager. 

John  Smith 

is 

a  man. 

Arturo  Rubio 

i  was 

the  assassin. 

Criteria:  Demonstrated  understanding  by  producing  an  acceptable  paraphrase. 

_ _ _ _ _ Score: 


1.2  Imperative  Sentences 


An  imperative  sentence  expresses  a  command  or  directive  and  commonly  consists  solely  of 
a  verb  phrase,  that  is,  there  is  no  overt  grammatical  subject.  The  verb  phrase  typically  has 
a  verb  in  base  form  followed  by  a  [NP]. 


Pattern: 

Examples: 


[Verb]  [NP]. 

List  the  departments. 

Show  the  salespersons. 

Display  the  NYS  map. 


Criteria:  Demonstrated  understanding  by  performing  the  commanded  action. 


Score: 


1.3  Interrogative  Sentences 
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An  interrogative  sentence  is  primarily  used  to  ask  questions.  For  more  information  on  inter- 
rogatives,  see  Section  II,  Interrogative  Sentences. 


Pattern: 

[WH-Word]  ([NP]) 

[BE-Verb]  [NP]? 

Examples: 

What 

are 

the  products? 

Which 

employees 

are 

women? 

Who 

are 

the  salespersons? 

Pattern: 

[WH-Word]  ([NP]) 

[DO-Verb]  [NP] 

[Verb]? 

Examples: 

What 

does 

John  Smith 

earn? 

What 

salary 

does 

John  Smith 

earn? 

Where 

does 

John  Smith 

work? 

What 

did 

the  guerrillas 

attack? 

Pattern: 

[BE-Verb] 

[NP] 

[Adjective]? 

Example: 

Is 

Mary  Smith 

experienced  ? 

Pattern: 

[BE-Verb] 

[NP] 

[NP]? 

Example: 

Is 

John  Smith 

a  salesperson  ? 

Pattern: 

[DO-Verb] 

[NP] 

[Verb] 

[NP]? 

Example: 

Does 

John  Smith 

have 

a  degree  ? 

Criteria:  Demonstrated  understanding  by  responding  with  at  least  the  requested  informa¬ 
tion. 


Score: 


2.  Simple  Determiners 

A  determinative  element  of  a  noun  phrase  determines  what  kind  of  reference  a  noun  phrase 
has:  definite  (“the”),  indefinite  ("a”,  “an”),  partitive  (“some”),  or  universal  (“all”).  Each 
type  of  reference  has  meaning.  For  example,  the  meaning  of  a  noun  phrase  with  a  definite 
determiner  ( “the”)  frequently  is  that  the  entity  expressed  by  the  noun  phrase  is  known  to 
both  speaker  and  hearer. 

The  purpose  of  this  section  is  to  test  for  acceptance  of  simple  determiners.  The  simple 
determiners  are  the  articles ,  which  include  the  definite  article  “the”,  and  the  indefinite 
articles  “a”  and  “an”. 

More  advanced  determinative  issues  are  covered  in  Section  III,  subsection  3,  Determinatives. 
Reference  issues  concerning  determinatives  are  covered  in  Section  X,  Reference.  Quantifier 
determinatives  are  tested  in  Section  VI,  Quantifiers. 

2.1  The  Indefinite  Article 

The  primary  use  of  the  indefinite  article  “a”  or  “an”  is  to  indicate  that  the  head  of  the  noun 
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phrase  in  which  it  is  the  determiner  denotes  an  individual  member  of  a  class,  but  which 
member  is  not  specified. 

The  noun  head  following  the  indefinite  articles  must  be  singular,  though  the  indefinite  article 
places  emphasis  on  the  type  of  entity  expresses  by  the  noun  phrase  and  not  the  number 
(singular)  of  the  entity  being  expressed.  Compare  the  following  two  sentences: 

Can  a  boy  carry  that  plank?  —  No,  but  a  man  can. 

Can  one  boy  carry  that  plank?  —  No,  but  two  boys  can. 

The  generic  sense  of  the  indefinite  article  will  be  covered  in  Section  X,  Reference. 

You  may  use  any  of  the  successful  basic  sentence  patterns  from  subsection  1  to  test  a  noun 
phrase  which  has  a  count  noun.  Replace  the  tt[NP]”  in  the  chosen  sentence  with  “[Indef 
Article]  [Noun].”  Below  are  some  suggested  sentence  patterns. 


Pattern: 

Example: 

[Verb] 

List 

[Indef  Article] 
a 

[Noun]. 

department. 

Pattern: 

Example: 

[BE-Verb] 

Is 

[NP] 

John  Smith 

[Indef  Article] 
a 

[Noun]. 

salesperson? 

Pattern: 

Example: 

[WH-Word] 

Who 

[BE-Verb] 

is 

[Indef  Article] 
an 

[Noun]. 

employee? 

Pattern: 

Examples: 

[NP] 

Terrorists 

Guerrillas 

[Verb] 

kidnapped 

attacked 

[Indef  Article] 
a 

a 

[Noun]. 

civilian. 

farm. 

Criteria:  Accepted  the  indefinite  article  and  produced  a  correct  response. 

- Score: 


2.2  The  Definite  Article 

The  primary  use  of  the  definite  article,  “the”,  is  that  it  signals  that  the  object  or  entity 
expressed  by  the  noun  head  should  be  familiar  to  the  speaker  and  hearer.  For  example, 
in  the  phrase  “the  manager  of  the  company”  it  is  supposedly  understood  which  manager 
is  being  referred  to,  either  because  there  is  only  one  manager,  or  because  the  manager  has 
already  been  mentioned  in  the  current  discourse. 

The  definite  article  may  also  be  used  with  a  singular  nominal  in  a  generic  sense  to  make  an 
assertion  about  a  whole  class,  equally  applicable  to  each  member  of  the  class. 

Section  X,  Reference  covers  the  different  types  of  usage  of  definite  articles  in  noun  phrases. 

The  noun  head  which  follows  the  definite  article  may  be  a  mass  noun  or  a  singular  or  plural 
count  noun. 

You  may  use  any  of  the  successful  basic  sentence  patterns  from  subsection  1  to  test  a  noun 
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phrase  which  has  a  count  noun.  Replace  the  “[NP]”  in  the  chosen  sentence  with  “[Def  Article] 
[Noun].”  Below  are  some  suggested  sentence  patterns. 


Pattern: 

[Verb] 

[Def  Article] 

[Noun]. 

Example: 

List 

the 

departments. 

Pattern: 

[WH-Word] 

[BE-Verb] 

[Def  Article] 

[Noun]. 

Example: 

Who 

are 

the 

employees? 

Pattern: 

[NP] 

[Verb] 

[Def  Article] 

[Noun]. 

Examples: 

Terrorists 

killed 

the 

Mayor. 

Guerrillas 

attacked 

the 

farm. 

Criteria:  Accepted  the  definite  determiner  and  produced  a  correct  response. 

_ Score: 


3.  Simple  Noun  Phrases 
3.1  Count  Nouns 

Count  nouns  denote  individual  countable  entities,  such  as  “chairman”,  “employee”,  “depart¬ 
ments”,  “children”,  etc.  A  simple  noun  phrase  can  consist  of  a  determiner  followed  by  a 
count  noun,  as  in  “an  employee”,  “ the  governor”,  or  a  count  noun  with  no  determiner,  as  in 
“employees”,  “terrorists”.  This  is  represented  as: 

Pattern:  ([Determiner])  [Count  Noun] 

Examples:  an  employee 

salespersons 

terrorists 

the  mayor 

You  may  use  any  of  the  successful  basic  sentence  patterns  from  subsection  1  to  test  a  noun 
phrase  which  has  a  count  noun.  Replace  the  “[NP]”  in  the  chosen  sentence  with  “([Deter¬ 
miner])  [Count  Noun].”  Below  are  some  suggested  sentence  patterns. 


Pattern: 

[Verb] 

([Determiner]) 

[Count  Noun] 

Example: 

List 

the 

departments. 

Pattern: 

[BE-Verb] 

[NP] 

([Determiner]) 

[Count  Noun] 

Example: 

Is 

John  Smith 

a 

salesperson  ? 

Pattern: 

[WH-Word] 

[BE-Verb] 

([Determiner]) 

[Count  Noun] 

Examples: 

Who 

are 

employees? 

Who 

are 

the 

salespersons? 
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Pattern:  [NP]  [Verb]  ([Determiner])  [Count  Noun] 

Examples:  Terrorists  kidnapped  the  mayor. 

Terrorists  kidnapped  civilians. 

Guerrillas  killed  the  security  guard. 

Criteria:  Demonstrated  understanding  by  responding  with  output  which  included  informa¬ 
tion  pertaining  to  the  object  or  objects  specified  by  the  count  noun. 

Score:  _ 


3.2  Proper  Nouns 

Proper  nouns  name  specific  persons,  places,  and  things  such  as  “John  Smith”,  “Peru”, 
“ Santa  Elena”,  etc.  A  noun  phrase  can  consist  of  a  proper  noun  with  no  determiner.  This 
is  represented  as: 

Pattern:  [Proper  Noun] 

Examples:  John  Smith 
San  Salvador 

You  may  use  any  of  the  successful  basic  sentence  patterns  from  subsection  1  to  test  a  noun 
phrase  consisting  of  a  proper  noun.  Replace  the  “[NP]”  in  the  chosen  sentence  with  “[Proper 
Noun]”.  Below  are  some  suggested  sentence  patterns. 


Pattern: 

[BE-Verb] 

[Proper  Noun] 

[NP] 

Example: 

Is 

John  Smith 

a  salesperson  ? 

Pattern: 

[Proper  Noun] 

[BE-Verb] 

[NP] 

Example: 

Mary  Jones 

is 

a  salesperson. 

Pattern: 

[NP] 

[Verb] 

[Proper  Noun] 

Examples: 

Terrorists 

kidnapped 

Arturo  Rubio. 

Guerrillas 

attacked 

President  Cerezo. 

Criteria:  Demonstrated  understanding  by  responding  with  output  which  included  informa¬ 
tion  pertaining  to  the  object  specified  by  the  proper  noun. 

_ Score: _ 


3.3  Mass  Nouns 

Mass  nouns  denote  an  undifferentiated  mass  or  continuum  such  a s  “experience” ,  “money”, 
“information”,  “work”,  etc.  A  simple  noun  phrase  can  consist  of  a  mass  noun  or  the  definite 
article  “the”  followed  by  a  mass  noun.  This  is  represented  as: 

Pattern:  ([Determiner])  [Mass  Noun] 

Examples:  data 

the  information 

You  may  use  any  of  the  successful  basic  sentence  patterns  from  subsection  1  to  test  a  noun 
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phrase  consisting  of  a  mass  noun  with  an  optional  determiner.  Replace  the  “[NP]”  in  the 
chosen  sentence  with  “([Determiner])  [Mass  Noun]”.  Below  are  some  suggested  sentence 
patterns.  (Note  that  although  we  have  not  yet  tested  prepositional  phrases  ([PP])  as  part  of 
a  noun  phrase,  they  may  be  helpful  in  forming  coherent  sentences.  See  Section  III  subsection 
1,  for  more  information  on  prepositional  phrases  as  postmodifiers  of  a  noun  phrase.) 


Pattern: 

[Verb] 

([Determiner]) 

[Mass  Noun] 

«pp]) 

Example: 

Show 

information 

on  Mary  Smith. 

Pattern: 

[WH-Word] 

[BE-Verb] 

([Determiner]) 

[Mass  Noun] 

«pp]) 

Example: 

What 

is 

the 

information 

about  salespersons? 

Pattern: 

Examples: 

[NP] 

Terrorists 

[Verb] 

destroyed 

([Determiner]) 

[Mass  Noun] 
information. 

([pp]) 

Terrorists 

The  guerrillas 

destroyed 

stole 

the 

information 

food. 

about  the  uprising. 

Criteria:  Demonstrated  understanding  by  responding  with  output  which  included  informa¬ 
tion  pertaining  to  the  entity  specified  by  the  mass  noun. 


Score: 


4.  Simple  Verb  Phrases 
4.1  Copular  Verb  Phrases 

A  simple  copular  verb  phrase  consists  of  a  form  of  the  BE- Verb  (am,  is,  are)  followed  by  a 
subject  complement ,  that  is,  an  adjective  or  a  noun  phrase  which  describes  the  subject  of  the 
sentence.  Thus  a  simple  type  of  copular  verb  phrase  consists  of: 

Pattern:  [Verb]  [Subject  Complement] 

[Verb]  [NP]  or  [Adjective] 

Examples:  is  intelligent 

is  male 

are  managers 

The  following  example  sentences  contain  copular  verb  phrases.  According  to  the  MUC  rules, 
templates  are  only  generated  for  reports  of  terrorist  activities.  For  a  MUC  example  we 
may  use  a  preliminary  sentence  to  express  a  terrorist  activity  in  order  to  generate  a  MUC 
template.  The  preliminary  sentence  may  follow  a  different  pattern  than  that  which  we  are 
testing. 


Pattern:  [NP]  [BE- Verb]  [Subject  Complement]. 

Examples:  Jane  Doe  is  a  manager. 

Jane  Doe  is  educated. 

Example:  (Preliminary  Sentence:  The  Mayor  of  Andahua  was  killed.) 

The  assassin  was  Arturo  Rubio. 

Example:  (Preliminary  Sentence:  The  guerrillas  killed  a  man.) 

The  murder  victim  is  a  civilian. 

Pattern:  [WH-Word]  ([NP])  [BE- Verb]  [Subject  Complement]  ? 

Examples:  Who  are  the  salespersons? 

Which  employees  are  female? 

Note  that  in  certain  Yes/ No  Questions  the  regular  sentence  order  is  altered  so  that  the  BE- 
Verb  precedes  the  subject  of  the  sentence  (the  first  noun  phrase).  The  following  example 
yes/no  questions  contain  copular  verb  phrases. 

Pattern:  [BE- Verb]  [NP]  [Subject  Complement] 

Examples:  Is  John  Smith  an  employee? 

Is  John  Smith  old? 

Criteria:  Demonstrated  understanding  that  the  subject  complement  defines  or  elaborates 
the  primary  noun  phrase. 

— - Score: _ 

4.2  Verb  Phrases  Involving  the  Auxiliary  Verb  DO 
Common  uses  of  DO  as  an  auxiliary  are: 

•  in  questions  and  other  constructions  that  involve  subject-verb  inversion,  where  the 
verb  is  in  the  simple  present  or  past  tense,  as  in  “ Did  John  finish  the  report?” 

•  in  emphatic  constructions  where  the  verb  is  simple  present  or  simple  past  tense,  as  in 
“the  chairman  does  want  you  to  attend  the  meeting.  ” 

The  structure  of  a  verb  phrase  which  includes  the  auxiliary  verb  DO  is  generally  as  follows: 

Pattern:  [VP] 

[DO-Verb]  [Verb]  ([NP]) 

Examples:  did  work 

does  earn  $30,000 

did  join  Calspan 

4.2.1  DO  Used  in  Interrogatives 

Certain  utterances  are  marked  as  questions  because  the  standard  sentence  order  ([NP]  [Aux] 
[Verbl  ([NP]))  is  altered.  In  some  of  these  questions,  the  auxiliary  DO  verb,  marked  with 
number  and  tense,  precedes  the  subject  of  the  sentence: 


Pattern:  [DO- Verb]  [NP]  [VP]? 

Examples:  Does  John  have  a  degree? 

Does  John  earn  $30,000? 

Did  the  guerrillas  attack  the  farm? 

In  other  questions,  a  WH-word,  or  WH-word  followed  by  a  noun  phrase  occurs  first,  followed 
by  the  auxiliary  DO  verb.  The  subject  and  verb  phrase  follow: 

Pattern:  [Wh-Word]  ([NP])  [DO- Verb]  [NP]  [VP]? 

Examples:  What  does  John  earn? 

When  did  John  join  Calspan? 

What  department  did  John  Smith  join? 

Who  did  the  guerrillas  kill? 


[VP]  ? 
earn? 

join  Calspan? 


Criteria:  Accepted  the  DO  verb  and  produced  a  correct  response. 


.  Score: 


4.2.2  The  Emphatic  DO 

The  auxiliary  verb  DO  precedes  the  main  verb  in  the  verb  phrase,  in  emphatic  DO  sentence 
constructions. 

Pattern:  [NP]  [DO- Verb]  [Verb]  [NP]  . 

Examples:  The  terrorists  did  kidnap  the  Andahua  mayor. 

The  terrorists  did  bomb  the  Santa  Elena  bank. 

The  guerrillas  did  attack  the  farm. 


Pattern:  [DO-Verb]  [Verb]  [NP] 

Example:  Do  show  the  employees. 

Criteria:  Accepted  the  DO  verb  and  produced  a  correct  response. 


[Verb] 

show 


[NP] 

the  employees. 


.  Score: 


4.3  Transitivity 

A  transitive  verb  is  followed  by  an  object  (commonly  called  the  direct  object),  as  in  “the 
manager  hired  John.”  An  intransitive  verb  is  not  followed  by  an  object,  for  example  the  verbs 
in  the  sentences  “Bill  studies ”,  “ the  chairman  resigned.”  A  ditransitive  verb  is  followed  by 
two  objects  (commonly  called  “the  indirect  object”  and  “the  direct  object”,  respectively), 
as  in  “Bill  gave  John  a  football ”,  “the  chairman  allowed  John  a  respite.”  Note  that  some 
verbs  are  in  more  than  one  of  these  three  categories. 

4.3.1  Simple  transitive  verb  phrase 

A  simple  transitive  verb  phrase  consists  of  a  transitive  verb  followed  by  a  noun  phrase: 
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Pattern:  [Verb  +trans]  [NP] 

Examples:  hired  John  Smith 

earn  $30,000 

kidnap  the  mayor 

You  may  use  any  of  the  basic  sentence  patterns  from  subsection  1  to  test  a  transitive  verb 
phrase.  Replace  the  “[VP]”  in  the  chosen  sentence  with  “[Verb  -{-trans]  [NP].”  Below  are 
some  suggested  sentences. 


Pattern: 

[Verb  *4" trans] 

[NP]- 

Examples: 

List 

the  departments. 

Pattern: 

[WH-Word] 

([NP]) 

[Verb  4-trans] 

[NP]? 

Examples: 

Who 

hired 

John  Smith? 

Which 

personnel 

earn 

$30,000? 

Pattern: 

[DO-Verb] 

[NP] 

[Verb  -(-trans] 

[NP]? 

Example: 

Does 

John  Smith 

have 

a  degree  ? 

Pattern: 

[NP] 

[Verb  +trans] 

[NP]. 

Examples: 

Terrorists 

kidnapped 

the  mayor. 

The  guerrillas 

attacked 

the  farm. 

Criteria:  Demonstrated  understanding  that  the  noun  phrase  following  the  verb  expresses  the 
object  of  the  event  or  relationship  specified  by  the  transitive  verb. 

_ Score: _ 


4.3.2  Simple  Intransitive  Verb  Phrase 

A  simple  intransitive  verb  phrase  consists  of  an  intransitive  verb,  that  is,  one  which  does  not 
take  an  object. 

Pattern:  [Verb  -I-intrans] 

Examples:  cleans 
runs 

You  may  use  any  of  the  basic  sentence  patterns  from  subsection  1  to  test  an  intransitive  verb 
phrase.  Replace  the  “[VP]”  in  the  chosen  sentence  with  “[Verb  +  intrans].”  Below  are  some 
suggested  sentence  patterns.  Remember  that  a  preliminary  sentence  may  be  given  prior  to 
a  test  input.  The  preliminary  sentence  may  follow  a  different  pattern  than  that  which  we 
are  testing. 
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Pattern: 

[WH-Word] 

([NP])  [Verb  +INTRANS] 

Examples: 

Which 

department  sells  ? 

Who 

hires  ? 

Pattern: 

[NP] 

[Verb  -f  -  intrans]  . 

Example: 

(Preliminary  Sentence:  Terrorists  kidnapped  the  mayor.) 

The  Mayor 

died. 

Example: 

(Preliminary  Sentence:  The  guerrillas  kidnapped  a  civilian.) 

The  murder  victim 

died. 

Criteria:  Demonstrated  understanding  that  the  subject  acts  or  acted  in  the  way  specified  by 
the  intransitive  verb. 


Score: 


4.3.3  Simple  Ditransitive  Verb  Phrase 

A  simple  ditransitive  verb  phrase  consists  of  a  ditransitive  verb  followed  by  two  noun  phrases, 
commonly  called  the  indirect  object  and  the  direct  object,  respectively: 

Pattern:  [Verb  +ditrans]  [NP]  [NP] 

Examples:  gave  John  a  raise 

shows  the  chairman  the  report 

You  may  use  any  of  the  basic  sentence  patterns  from  subsection  1  to  test  an  ditransitive  verb 
phrase.  Replace  the  “[VP]”  in  the  chosen  sentence  with  “[Verb  +dithans]  [NP]  [NP].”  Below 
are  some  suggested  sentences.  Remember  that  a  preliminary  sentence  may  be  given  prior  to 
a  test  input.  The  preliminary  sentence  may  follow  a  different  pattern  than  that  which  we 
are  testing. 


Pattern: 

[Verb  -|-ditrans] 

[NP] 

[NP]. 

Example: 

Show 

me 

the  departments. 

Pattern: 

[WH-Word] 

[Verb  -f-DiTRANs] 

[NP] 

[NP]? 

Examples: 

Who 

gave 

John  Smith 

a  raise? 

Who 

shows 

the  chairman 

the  report? 

Pattern: 

[NP] 

[Verb  +DtTRANs] 

[NP] 

[NP] 

Example: 

Terrorists  sent 

(which  exploded). 

the  mayor 

a  letter  bomb 

Example: 

(Preliminary  Sentence:  The  government  suspects  the  URNG 

guerrillas  in  the  bombing  at  the  Government  house.) 

The  Mayor 

showed. 

a  journalist 

the  burned  building. 

Criteria:  Demonstrated  understanding  that 

•  the  second  noun  phrase  following  the  verb  expresses  the  object  of  the  verb’s  action 


1-14 


•  the  first  noun  phrase  following  the  verb  expresses  the  recipient. 

Sc/wv  _ 

4.4  Voice 

Verbs  of  the  English  language  have  two  voices:  active  and  passive.  Most  propositions  can  be 
expressed  using  either  voice.  All  the  sentences  in  the  previous  sections  are  given  in  the  active 
voice,  which  is  unmarked.  A  passive  verb  phrase  is  formed  with  the  BE-verb,  used  as  an 
auxiliary,  followed  by  the  -ed  participle  form  of  the  main  verb,  called  the  “ passive  participle” 
followed  by  an  optional  noun  phrase  and  an  optional  by-phrase  (the  word  “by”  followed  by 
a  [NP]): 

Pattern:  [BE-Verb]  [Verb  +passive]  ([NP])  (by  [NP]) 

Examples:  is  hired 

was  sold  a  book  by  John 

were  given  notice 

was  kidnapped  by  terrorists 

Any  sentence  which  includes  a  transitive  or  ditransitive  verb  phrase  can  be  expressed  using 
either  the  active  or  passive  voice.  An  active  voice  sentence  can  be  converted  into  a  passive 
voice  sentence,  and  vice  versa.  The  simple  active-passive  correspondence  can  be  expressed 
by  the  following  rule: 

[NP]j  [Verb  -f  active]  [NP]2  maps  into  [NP]2  [BE-Verb]  [Verb  +passive]  by  [NP]j 

Note  that  since  the  by-phrase  in  the  passive  is  optional,  the  agent  of  the  expressed  action  is 
not  always  present. 

The  following  pairs  of  sentences  provide  examples  of  active-passive  sentences.  In  each  pair 
of  sentences,  both  sentences  express  the  same  proposition. 

Active  Example:  John  Smith  sells  CPUs. 

Passive  Example:  CPUs  are  sold  by  John  Smith. 

Active  Example:  What  does  John  Smith  sell  ? 

[Alternative  Version  of  the  Active  Example: 

John  Smith  sells  what  ?  ] 

Passive  Example:  What  is  sold  by  John  Smith? 

Active  Example:  Does  John  Smith  sell  printers  ? 

[Alternate  Version  of  the  Active  Example: 

John  Smith  sells  printers  ? 

Passive  Example:  Are  printers  sold  by  John  Smith? 

[Alternate  Version  of  the  Passive  Example: 

Printers  are  sold  by  John  Smith?  ] 

Active  Example:  The  manager  gave  John  Smith  a  promotion. 
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Passive  Example:  John  Smith  was  given  a  promotion  by  the  manager. 

Active  Example:  Terrorists  kidnapped  the  governor. 

Passive  Example:  The  governor  was  kidnapped  by  terrorists. 

Active  Example:  The  guerrillas  attacked  the  farm. 

Passive  Example:  The  farm  was  attacked  by  the  guerrillas. 

4.4.1  Active  Voice 

In  this  subsection  we  will  test  an  active  sentence  which  we  will  later  passivize.  Choose  an 
active  sentence  with  a  transitive  or  ditransitive  verb  for  testing.  You  may  also  use  any 
sentence  which  has  been  successfully  tested  and  enter  it  below. 

Criteria:  Demonstrated  understanding  that  the  object  was  acted  upon  in  the  way  specified 
by  the  verb. 

_ Score: _ 


4.4.2  Passive  Voice 

Passivize  the  sentence  which  you  used  in  subsection  4.4.1. 

Criteria:  Demonstrated  understanding  that  the  subject  was  acted  upon  in  the  way  specified 
by  the  verb. 

_ Scoer: _ 
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II.  INTERROGATIVE  SENTENCES 


An  interrogative  sentence  expresses  a  question.  In  this  section,  you  should  use  simple  com¬ 
ponent  phrases  (i.e.,  simple  [NP]s  and  (VP]s).  In  this  set  of  interrogative  sentences,  we  will 
be  testing  for  the  capability  to  handle  the  basic  sentence  structure  and  we  do  not  want  to 
make  the  component  phrases  at  all  complicated.  These  complications  will  come  later. 

According  to  the  MUC  rules,  templates  are  only  generated  for  reports  of  terrorist  activities. 
For  a  MUC  example  we  may  use  a  preliminary  sentence  to  express  a  terrorist  activity  in 
order  to  generate  a  MUC  template.  The  preliminary  sentence  may  follow  a  different  pattern 
than  that  which  we  are  testing. 

I.  What-questions 

1.1  What  as  a  pronoun. 

Create  a  test  sentence  that  starts  with  the  word  “what”  used  as  an  interrogative  pronoun 
(and  not  as  a  determiner  followed  by  a  noun  -  this  is  tested  in  the  next  item). 

Pattern:  What  [BE- Verb]  [NP]  ? 

Examples:  What  are  the  departments? 

What  are  the  products? 

Example:  (Preliminary  Sentence:  The  guerrillas  killed  five  civilians.) 

What  are  the  fatalities? 

Pattern:  What  [DO- Verb]  [NP]  [VP]  ? 

Example:  What  does  John  Smith  earn? 

Example:  (Preliminary  Sentence:  The  URNG  guerrillas  attacked  the  farm.) 

What  did  the  guerrillas  attack? 

Criteria:  Provided  the  information  requested. 

_ Score: _ 

1.2  What  as  a  determiner. 

Create  a  test  sentence  that  starts  with  the  word  “what”  followed  by  a  noun  or  noun  phrase 
(reference  to  a  type  of  object),  (e.g.,  “ what  salary”)  so  that  the  word  “what”  is  used  as  a 
determiner. 
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Pattern:  What 

(NP] 

[Verb] 

[NP]  ? 

Examples:  What 

personnel 

are 

women? 

What 

personnel 

are 

engineers? 

Pattern:  What 

[NP] 

[BE- Verb] 

[Adjective]  ? 

Examples:  What 

departments 

are 

small? 

What 

salespersons 

are 

married? 

Example:  (Preliminary  Sentence:  The  guerrillas  attacked 

the  Santo  Tomas  farm.) 

What 

farm 

was 

damaged? 

Pattern:  What 

[NP] 

[DO-Verb] 

[NP] 

[VP]  ? 

Examples:  What 

salary 

does 

John  Smith 

earn? 

What 

department 

did 

John  Smith 

join? 

What 

experience 

does 

John  Smith 

have? 

Example:  (Preliminary  Sentence:  URNG  guerrillas  attacked  the  Santo  Tomas  farm 

What 

farm 

did 

guerrillas 

attack? 

Criteria:  Restricted  its  response  to  information  about  objects  in  the  class  or  class  type  named 
by  the  expression  “what  [. NP ]”  (e.g.,  “what  personnel”)  and  restricted  its  response  by  the 
predicate  expressed  in  the  rest  of  the  sentence. 

_ Score:  - 


2.  Who-questions. 

Create  a  test  sentence  that  starts  with  the  word  “who”  used  as  an  interrogative  pronoun. 


Pattern: 

Who 

[Verb] 

[NP]  ? 

Examples: 

Who 

is 

a  department  head  ? 

Who 

are 

the  salespersons  ? 

Who 

hired 

John  Smith? 

Example: 

(Preliminary  Sentence:  URNG  guerrillas 

attacked  the  Santo  Tomas  farm.) 

Who 

attacked 

the  farm? 

Pattern: 

Who 

[DO-Verb] 

[NP] 

[VP]  ? 

Examples: 

Who 

did 

Ted  Black 

hire? 

Who 

does 

the  sales  department 

employ? 

Example:  (Preliminary  Sentence:  The  guerrillas  attacked  President  Cerezo.) 

Who  did  the  guerrillas  attack? 


Criteria:  Responded  by  identifying  the  person(s)  that  satisfies  the  criteria  expressed. 


3.  Where-questions. 

Create  a  test  sentence  that  starts  with  the  word  “where”  and  requests  locative  information. 


Pattern: 

Where 

[BE-Verb] 

[NP]  ? 

Examples: 

Where 

is 

the  Computer  Science  Department  ? 

Where 

is 

the  California  branch  ? 

Example: 

(Preliminary  Sentence:  The  guerrillas  attacked  a  farm  in  Flores.) 

Where 

is 

the  farm? 

Pattern: 

Where 

[DO-Verbj 

[NP]  [VP]  ? 

Examples: 

Where 

does 

Ted  Black  live? 

Where 

does 

John  Smith  work? 

Example: 

(Preliminary  Sentence:  The  guerrillas  attacked  a  farm  in  Flores.) 

Where 

did 

the  guerrillas  attack? 

Criteria:  Responded  with  the  requested  LOCATIVE  information? 

_ _ _ Score: 


4.  When-questions. 

Create  a  test  sentence  that  starts  with  the  word  “when”  and  requests  temporal  information. 


Pattern:  When 

[BE-Verb] 

[NP]  ? 

Examples:  When 

is 

Thanksgiving  ? 

When 

is 

William’s  birthday  ? 

Example:  (Preliminary  Sentence:  Guerrillas  attacked 

a  farm  on  February  2.) 

When 

was 

the  attack? 

Pattern:  When 

[DO- Verb] 

[NP] 

[VP]  ? 

Examples:  When 

did 

Ted  Black 

join  Xidec? 

When 

did 

John  Smith 

graduate? 

Example:  (Preliminary  Sentence:  Guerrillas  attacked 

a  farm  on  February  2.) 

When 

did 

the  guerrillas 

attack? 

Criteria:  Responded  with  the  requested  TEMPORAL  information. 


Score: 
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5.  Which-questions. 

Create  a  test  sentence  that  starts  with  the  word  “which”  (as  a  determiner)  followed  by  a 
noun  or  noun  phrase  so  that  information  is  requested  about  members  of  the  class  specified 
by  the  noun  phrase  (e.g.,  “which  personnel”). 


Pattern: 

Which 

[NP]  [Verb] 

[NP]  ? 

Examples: 

Which 

Which 

personnel  are 
personnel  are 

engineers? 

women? 

Pattern: 

Which 

[NP]  [Verb] 

[NP]  ? 

Examples: 

Which 

Which 

personnel  have 
personnel  earn 

degrees  ? 
$30,000  ? 

Example: 

(Preliminary  Sentence:  URNG  guerrillas  attacked  the  Santo  Tomas  farm.) 
Which  guerrillas  attacked  the  farm? 

Pattern: 

Which 

[NP]  [BE-Verb] 

[Adjective]  ? 

Examples: 

Which 

Which 

departments  are 
salespersons  are 

small? 

male? 

Pattern: 

Which 

[NP]  [DO-Verb] 

[NP] 

[VP]  ? 

Examples: 

Which 

department  did 

John  Smith 

join? 

Which 

degree  did 

John  Smith 

earn? 

Example:  (Preliminary  Sentence:  URNG  guerrillas  attacked  the  Santo  Tomas  farm.) 

Which  farm  did  URNG  guerrillas  attack? 

Criteria:  Restricted  its  response  to  information  about  objects  in  the  class  or  class  type  named 
by  the  expression  “which  [NP]”  ( e.g “which  personnel”)  and  restricted  its  response  by  the 
predicate  expressed  in  the  rest  of  the  sentence. 

— — _ _ _ Score: _ 

6.  How-questions. 

Create  a  test  sentence  that  starts  with  the  word  “how ”  followed  by  an  adjective  or  an 
adverb  so  that  information  is  requested  about  the  property  specified  by  the  expression  “how 
[Adjective]”  or  “how  [Adverb]”. 


Pattern:  How 

[Adjective] 

[BE-Verb] 

[NP]  ? 

Examples:  How 

old 

is 

John  Smith  ? 

How 

large 

is 

the  sales  department  ? 

How 

experienced 

is 

John  Doe  ? 

Example:  (Preliminary  Sentence:  The  URNG  guerrillas  burned  the  Santo  Tomas  farm.) 

How  extensive  was  the  damage? 
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Pattern:  How  [Adverb]  [BE- Verb]  [NP]  ? 

Examples:  How  often  is  payday  ? 

How  soon  is  Ted  William’s  birthday  ? 

Example:  (Preliminary  Sentence:  Guerrillas  attack  the  Santo  Tomas  farm  daily.) 

How  often  are  the  attacks? 

Criteria:  Responded  with  information  about  the  property  specified  by  the  phrase  “how 
l Adjective ]”  or  “i how  [Adverb]”. 

_  Srnrp-  _ 


7.  Wh-Word  in  Prepositional  Phrase 

Create  a  test  question  that  uses  a  Wh-word  or  Wh-phrase  as  the  object  of  a  prepositional 
phrase.  Examples  of  such  prepositional  phrases  are  “for  whom”,  “by  whom”,  “in  which 
year”,  “in  which  department”.  The  prepositional  phrase  can  appear  at  the  beginning  of 
the  question  OR  the  preposition  and  its  following  noun  phrase  can  be  separated  with  the 
preposition  appearing  at  the  end  of  the  sentence  with  its  object  noun  phrase  at  the  beginning 
of  the  sentence.  Sentence  patterns  with  examples  are  shown  below.  See  the  glossary  for  a 
listing  of  prepositions. 


Pattern: 

[Preposition] 

[Wh-Phrase] 

[BE- Verb]/ 
[DO-Verb] 

[NP] 

[Verb]  ? 

Examples: 

For 

whom 

does 

John  Smith 

work  ? 

In 

which  department 

is 

Mary  Jones 

employed  ? 

In 

which  year 

was 

John  Smith 

born  ? 

Example:  (Preliminary  Sentence:  Guerrillas  attacked  the  Santo  Tomas  farm  on  February  2.) 


On  which  day 

was 

the  attack? 

Pattern: 

[Wh-Phrase] 

[BE-Verb]/ 

[DO-Verb] 

[NP] 

[Verb] 

[Preposition]  ? 

Examples: 

Whom 

does 

John  Smith 

work 

for  ? 

Which  department 

is 

Mary  Jones 

employed 

in  ? 

Which  year 

was 

John  Smith 

born 

in  ? 

Criteria:  Responded  with  information  about  the  object  or  entity  specified  by  the  noun  phrase 
that  belongs  with  the  preposition.  For  example,  if  asked  “Which  department  is  Mary  Jones 
employed  in?”,  the  system  responded  with  information  about  the  appropriate  department. 

_ _ _ Score: _ 


8.  Yes/No-Questions. 

Create  a  test  sentence  to  which  a  “yes/no”  answer  is  expected,  possibly  with  additional 
information.  The  common  forms  of  such  questions  are  provided  below. 


Pattern:  [BE- Verb]  [NP]  [NP]  ? 

Examples:  Is  the  President  John  Doe  ? 

Is  John  Smith  a  salesman  ? 


Example: 


(Preliminary  Sentence:  Guerrillas  attacked  the  presidential  farm.) 
Was  the  president  a  victim? 


Pattern:  [BE- Verb] 

Examples:  Is 

Is 


[NP]  [Adjective]  ? 

the  Computer  Science  large  ? 
Department 

Mary  Smith  experienced  ? 


Pattern:  [DO-Verb]  [NP] 

Examples:  Does  John 

Does  John 


[VP]  ? 

have  a  degree  ? 
earn  $30,000  ? 


Example: 


(Preliminary  Sentence:  URNG  guerrillas  attacked  the  presidential  farm.) 
Did  guerrillas  attack  the  presidential  farm? 


Criteria:  Responded  either  with  a  “yes/no”  answer  (additional  information  is  also  accept¬ 
able)  or  with  a  statement  (without  the  “yes/no”)  that  answered  the  question. 

_ Score: - 


9.  Why-questions,  (t.b.d.) 
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in.  NOUN  PHRASES 


A  noun  phrase  consists  of  four  parts: 

([Determinative])  ([Premodification])*  [Head]  ([Postmodification])* 

of  which  any,  but  the  Head,  may  be  omitted,  and  there  may  be  more  than  one  instance  of 
premodification  and  postmodification  (indicated  by  the  “*”).  Henceforth,  we  will  use  the 
notation  “[Det]”  to  denote  the  determinative  element. 

A  noun  phrase  expresses  an  object  or  entity,  which  may  be  either  abstract  or  concrete. 

1.  Prepositional  Phrase  as  Postmodifier  in  a  Noun  Phrase 

A  prepositional  phrase,  denoted  “[PP]”,  consists  of  a  preposition  followed  by  a  complement, 
which  is  typically  a  noun  phrase.  Some  prepositional  phrases  are: 

Pattern:  [PP] 

[Preposition]  [NP] 

Examples:  in  the  department 

for  Joe  Smith 

on  December  12 

A  noun  phrase  can  consist  of  a  simpler  noun  phrase  followed  by  a  prepositional  phrase  as  the 
postmodification  element.  The  prepositional  phrase  helps  define  the  referent  (who  or  what 
is  being  referred  to)  of  the  preceding  noun  phrase  which  it  postmodifies.  A  noun  phrase  can 
consist  of  the  following  structure: 

Pattern:  [NP] 

[NP]  [PP] 

Examples:  the  name  of  the  ship 

the  director  of  the  project 

the  employees  in  New  York 

the  farm  of  President  Cerezo 

a  peasant  in  Flores 

Prepositional  phrases  can  also  serve  as  adverbials  and  complements  of  verbs  or  adjectives. 
These  uses  of  prepositional  phrases  are  covered  in  later  sections  of  this  procedure. 

The  following  is  a  fairly  complete  list  of  the  prepositions: 


as 

at 

but 

by 

down 

for 

from 

in 

near 

of 

off 

on 

out 

past 

per 

since 

through 

till 

to 

up 

with 

about 

above 

across 

after 

against 

along 

among 

around 

atop 

before 

behind 

below 

beneath 

beside 

between 

beyond 

during 

except 

inside 

into 

onto 

outside 

over 

throughout  toward 

under 

underneath 

until 

upon 

within 

without 

I1I-1 


You  may  use  any  of  the  successful  sentence  patterns  from  Sections  I  or  II  to  test  the  system’s 
ability  to  process  PPs.  Replace  the  “[NP]”  in  the  chosen  sentence  with  “[NP]  [PP].”  Below 
are  some  suggested  patterns.  The  prepositional  phrase  is  shown  explicitly  in  the  following 
examples. 


Pattern: 

Example: 

[Verb] 

List 

[NP] 

the  employees 

[pp] 

in  New  York. 

Pattern: 

Example: 

[BE-Verb] 

Is 

[NP] 

John  Smith 

[NP] 

a  salesman 

[pp] 

in  Chicago  ? 

Pattern: 

Example: 

[WH-Word] 

Who 

[BE-Verb] 

are 

[NP] 

the  salespersons 

[pp] 

in  Chicago? 

Pattern: 

Examples: 

[NP] 

Terrorists 

Guerrillas 

Guerrillas 

[Verb] 

kidnapped 

attacked 

killed 

[NP] 

the  mayor 
the  farm 
a  peasant 

[PP] 

of  Andahua. 
of  President  Cerezo. 
in  Flores. 

Criteria:  Demonstrated  understanding  of  the  qualification  imposed  on  the  noun  phrase  by 
the  prepositional  phrase  which  immediately  follows  it  as  postmodifier. 


Score: 


2.  The  Noun  Head 

2.1  Nouns 

2.1.1  Count  Noun  (Covered  in  Section  I,  Basic  Sentences) 

2.1.2  Proper  Noun  (Covered  in  Section  I,  Basic  Sentences) 

2.1.3  Mass  Nouns  (Covered  in  Section  I,  Basic  Sentences) 

2.2  Nominals 

Nominals  are  non-noun  parts  of  speech  used  as  the  head  of  a  noun  phrase. 

2.2.1  Adjective  Nominal 

An  adjective  nominal  consists  of  an  optional  determiner  followed  by  an  adjective  used  as 
the  head  of  the  noun  phrase.  Examples:  “ the  elderly”,  “the  poor”,  “the  youngest”,  “the 
smartest”. 

You  may  use  any  of  the  successful  sentence  patterns  from  Sections  I  or  II  to  test  an  adjective 
nominal.  Replace  the  “[NP]”  in  the  chosen  sentence  with  u([Det])  [Adj.Nom.]”.  We  use 
the  notation  “[Det]”  to  denote  the  determinative  element  and  “[Adj.Nom.]”  to  denote  the 
adjective  nominal. 
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Below  are  some  suggested  patterns. 


Pattern: 

[Verb] 

([Det]) 

[Adj.Nom.] 

«pp]) 

Example: 

List 

the 

oldest 

in  the  New  York  branch. 

Pattern: 

[WH-Word] 

[BE-Verb] 

([Det]) 

[Adj.Nom.] 

([pp]) 

Example: 

Who 

is 

the 

oldest 

in  Chicago? 

Pattern: 

[NP] 

[Verb] 

([Det]) 

[Adj.Nom.] 

([pp]) 

Example: 

Terrorists 

killed 

the 

elderly 

in  Usulutan. 

Pattern: 

([Det]) 

[Adj.Nom.] 

«pp]) 

[Verb] 

[NP]- 

Example: 

The 

poor 

in  Usulutan 

attacked 

the  Government  House 

Criteria:  Demonstrated  understanding  by  interpreting  the  adjective  nominal  as  expressing 
an  entity  or  object. 

_ Score: _ 


2.2.2  Passive  Participle  as  Nominal 

The  passive  participle ,  denoted  by  [Participle  (-ed)],  is  a  form  of  a  verb  with  the  u-ed”  or 
“-en”  ending,  as  in  “hired”,  “arrested” ,  or  “spoken”,  or  certain  irregular  verbs  such  as  “sung”. 

A  passive  participle  nominal  consists  of  an  optional  determiner  followed  by  a  passive  partici¬ 
ple  used  as  the  head  of  the  noun  phrase.  Examples:  “ the  employed”,  “the  injured”. 

You  may  use  any  of  the  successful  sentence  patterns  from  Sections  I  or  II  to  test  a  passive 
participle  nominal.  Replace  the  “[NP]”  in  the  chosen  sentence  with  “([Det])  [Participle 
(-ed)]”.  Below  are  some  suggested  patterns. 


Pattern: 

[Verb]  ([Det]) 

[Participle  (-ed)] 

([PP]) 

Example: 

List  the 

employed 

in  the  New  York  branch. 

Pattern: 

[WH-Word] 

[BE-Verb]  ([Det]) 

[Participle  (-ed)] 

([pp]) 

Example: 

Who 

are  the 

employed 

in  Chicago? 

Pattern: 

[NP] 

[Verb]  ([Det]) 

[Participle  (-ed)] 

([pp]) 

Example: 

(Preliminary  Sentence:  Twelve  men 

were  injured  in  an 

explosion.) 

Terrorists 

killed  the 

injured. 

Example: 

The  result  of 

was  one 

killed. 

the  attack 


Criteria:  Demonstrated  understanding  by  interpreting  the  passive  participle  nominal  as 
expressing  an  entity  or  object. 
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Score: 


2.2.3  Progressive  Participle  as  Nominal 

The  progressive  participle ,  denoted  by  [Participle  (-ing)],  is  a  form  of  a  verb  with  an  “-ing” 
ending,  as  in  “hiring ”,  and  “consulting” . 

A  progressive  participle  nominal  consists  of  an  optional  determiner  followed  by  a  progressive 
participle  used  as  the  head  of  the  noun  phrase.  Examples:  “the  hiring”,  “the  selling”. 

You  may  use  any  of  the  successful  sentence  patterns  from  Sections  I  or  II  to  test  a  progressive 
participle  nominal.  Replace  the  “[NP]”  in  the  chosen  sentence  with  u([Det])  [Participle  (- 
ing)]”.  Below  are  some  suggested  patterns. 


Pattern: 

[WH-Word]  [DO-Verb] 

([Det])  [Participle  (-ing)] 

([PP]) 

Examples: 

Who 

does 

the  selling 

in  Chicago? 

Who 

does 

the  hiring 

in  Chicago? 

Pattern: 

([Det]) 

[Participle  (-ing)]  ([PP]) 

[VP]. 

Examples: 

The 

hiring 

is  performed  by 
John  Smith. 

The 

bombing 

of  the  Government  House 

killed  20  civilians. 

The 

burning 

of  the  presidential  farm 

caused  the  death 
of  one  guard. 

Criteria:  Demonstrated  understanding  by  interpreting  the  progressive  participle  nominal  as 
expressing  an  entity  or  object. 

_ Score: - 


3.  Determinatives  in  More  Detail 

A  determinative  element  of  a  noun  phrase  determines  what  kind  of  reference  a  noun  phrase 
has:  definite  (“the”),  indefinite  (“a”,  “an”),  partitive  (“some”),  or  universal  (“all”).  Each 
type  of  reference  has  meaning.  For  example,  the  meaning  of  a  noun  phrase  with  a  definite 
determiner  (  “the”)  frequently  is  that  the  entity  expressed  by  the  noun  phrase  is  known  to 
both  speaker  and  hearer. 

The  determinative  element  contains  up  to  three  elements  (see  table  below).  We  can  distin¬ 
guish  three  classes  of  determiners, based  on  their  position  in  the  determinative.  These  are 
the  predeterminers,  the  central  determiners ,  and  the  the  postdeterminers. 

The  predeterminers  include  “all”,,  “both”,  “half”,  the  multipliers  ( “double”,  “twice”,  etc.), 
the  fractions  (“one-third”,  “one-fifth”,  etc.),  “such”,  and  “what”.  The  central  determin¬ 
ers  include  the  articles  (“the”,  “a”,  “an”),  the  demonstrative  determiners  (“this”,  “that”, 
“ these ”,  “those”),  the  possessive  determiners  (“my”,  “our”,  “your”,  “his”,  “her”,  “its”, 
“their”),  certain  quantifiers,  and  so  on.  The  postdeterminers  include  the  cardinal  numerals, 
ordinal  numerals,  and  a  small  set  of  quantifiers  (i.e.,  “many”,  “few”,  “ several ”  with  plural 
count  nouns  and  “much”  and  “little”  with  non-count  nouns). 

In  this  section  on  determinatives,  we  will  use  “(NP  — det]”  to  denote  a  complete  noun  phrase 
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Determinative 

Predeterminer 

Central  Determiner 

Postdeterminer 

all/both/half 

the 

a 

few 

such 

a/an 

twice/double 

this/that 

these/ those 
every /each 
some/ any 
no 

three 

one-third 

my/our/your/his/its/ their 
which /what/  whose 
either/neither 

minus  the  determinative  we  are  testing. 

3.1  Predeterminers 

3.1.1  Quantifier  Predeterminers  (Covered  in  Section  VI,  Quantifiers) 

3.1.2  Multipliers,  Fractions  (Covered  in  Section  VI,  Quantifiers) 

3.1.3  WHAT,  SUCH  (t.b.d.) 

3.2  Central  Determiners 

3.2.1  Articles  (Covered  in  Section  I,  Basic  Sentences.) 

3.2.2  Demonstrative  Determiners  (t.b.d.) 

3.2.3  Quantifier- type  (Covered  in  Section  VI,  Quantifiers) 

3.2.4  Quantitative  Determiners  (t.b.d.) 

3.2.5  Possessive  Determiners  (Covered  in  Section  X,  Reference.) 

3.2.6  Relative  Determiners 

The  relative  determiners  “whose”  and  “which”  are  determiners  which  are  used  to  introduce 
relative  clauses.  In  the  following  two  examples  the  relative  clause  is  enclosed  by  brackets  for 
ease  of  identification: 

The  lady  [whose  car  you  hit]  was  upset 

Call  again  at  11  [by  which  time  the  meeting  will  be  over]. 


Relative  determiners  are  covered  in  the  subsection  on  relative  clauses. 

3.2.7  Interrogative  Determiners  (Covered  in  Section  II,  Interrogative  Sentences) 
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3.2.8  Wh- Determiners 

The  following  items  test  the  Wh-Determiners  “whatever”,  “whichever”,  and  “whosever”. 


3.2.8. 1  Indefinite  Reference  ■ WHATEVER ) 


Pattern: 

List 

whatever 

[NP  —det] 

[VP). 

Example: 

List 

whatever 

departments 

have  10  employees. 

Pattern: 

[NP] 

[Verb] 

whatever 

[NP  —det] 

[VP]. 

Examples: 

Terrorists 

bombed 

whatever 

buildings 

stood. 

Guerrillas 

stole 

whatever 

food 

was  available. 

Criteria:  Demonstrated  understanding  by  interpreting  the  determiner  “whatever”  as  restrict¬ 
ing  the  [NP  — det]  to  persons  or  objects  having  the  attributes  expressed  by  the  verb  phrase 
immediately  following. 

_ Score: _ 


3.2.8.2  Definite  Reference  ( WHICHEVER) 


Pattern: 

List 

whichever 

[NP  —det] 

[VP]. 

Example: 

List 

whichever 

departments 

have  10  employees. 

Pattern: 

[NP] 

[Verb] 

whichever 

[NP  —det] 

[VP]. 

Examples: 

Terrorists 

bombed 

whichever 

buildings 

stood. 

Guerrillas 

attacked 

whichever 

farms 

belonged  to. 
the  president. 

Criteria:  Demonstrated  understanding  by  interpreting  the  determiner  “whichever”  as  re¬ 
stricting  the  [NP  —det]  to  persons  or  objects  having  the  attributes  expressed  by  the  verb 
phrase. 

— _ Score:  _ 


3.2.8. 3  Personal  Reference  ( WHOSEVER )  (t.b.d) 

3.3  Postdeterminers 

3.3.1  Cardinal  Numbers  (Covered  in  Section  VI,  Quantifiers) 

3.3.2  Ordinal  Numbers  (Covered  in  Section  VI,  Quantifiers) 

3.3.3  General  Ordinals  (t.b.d.) 

3.3.4  Closed-class  Postdeterminers  (t.b.d.) 

3.3.5  Open-class  Postdeterminers  (t.b.d.) 


3.4  Determinative  Combinations  (Covered  in  Section  VI,  Quantifiers) 

4.  Premodification 

The  premodifier  of  a  noun  phrase  may  contain  adjectives,  adverbs  modifying  these  adjectives, 
passive  participles,  progressive  participles ,  and  nouns.  The  accompanying  table  shows  the 
noun  phrase  structure  with  examples. 


!  Noun  Phrase  Structure  without  Postmodification 

Deter¬ 

minative 

Premodification 

Head 

Precentral 

Central 

Postcentral 

Prehead 

all  the 

our 

a 

the 

very 

very 

extremely 

large 

tall 

young 

newly  hired 
costly 

faded  yellow 
educated 

tourist 

research 

Gothic 

computer 

attraction 

engineers 

debt 

building 

scientist 

The  ordering  of  the  premodification  group  is  generally  as  follows: 

1.  Precentral  -  intensifies,  down- toners,  or,  more  generally,  adverbs  which  modify  the 
central  premodifier  (e.g.,  “very” ,  “ more ”,  “often”) 

2.  Central  -  an  adjective  head  (e.g.,  “great”,  “big”,  “small”,  “round”) 

3.  Postcentral  -  the  passive  participle  (e.g.,  “hired”,  “paid”),  the  progressive  participle 
(e.g.,  “talking”,  “working”),  color  adjectives  (e.g.,  “yellow”) 

4.  Prehead  -  the  most  nominal  elements,  such  as  denominal  adjectives  (e.g.,  “Dutch”, 
“brick”,  “stone”,  “Gothic”),  nouns 

4.1  Minimal  Relation  (t.b.d.) 

4.2  Central  Premodifiers 

Central  premodifiers  in  a  noun  phrase  express  attributes  of  the  entity  expressed  by  the 
subsequent  portion  of  the  noun  phrase.  In  general,  the  central  premodifiers  are  gradable 
adjectives.  Gradable  adjectives  express  properties  for  which  there  is  an  associated  scale  with 
the  property  ranging  from  low  intensity,  strength,  or  amount  to  high  intensity,  strength,  or 
amount.  For  example,  the  adjective  “hungry”  expresses  the  property  of  having  hunger,  which 
can  be  measured  on  a  scale  ranging  from  being  a  little  hungry  to  being  very  hungry(f). 

In  the  following  subsections  we  will  test  noun  phrases  of  the  form: 

[Det]  (Central  Premodifier]  [Noun] 
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4.2.1  Simple  Adjectives 

In  this  section  we  use  an  adjective  as  central  premodifier.  The  noun  phrase  we  are  testing 
here  has  the  pattern  “(Detj  [Adjective]  [Noun]”.  You  may  replace  the  [NP]  in  any  of  the 
basic  sentence  patterns  with  this  pattern.  Below  are  some  suggested  sentence  patterns. 


Pattern: 

List 

[Det] 

[Adjective] 

[Noun]. 

Example: 

List 

the 

top 

salary. 

Pattern: 

[WH-Word] 

[BE- Verb] 

[Det] 

[Adjective] 

[Noun]  ? 

Example: 

Who 

are 

the 

new 

employees? 

Pattern: 

[Det] 

[Adjective] 

[Noun] 

[VP]. 

Example: 

The 

liberal 

president 

was  killed  by  terrorists. 

Pattern: 

[NP] 

[Verb] 

[Det] 

[Adjective] 

[Noun]. 

Examples: 

Terrorists 

kidnapped 

the 

new 

president. 

Guerrillas 

burned 

the 

new 

facilities. 

Criteria:  Demonstrated  understanding  by  correctly  interpreting  the  adjective  as  qualifying 
the  noun  in  accordance  with  the  attribute  expressed  by  the  adjective. 

_ Score: _ 


4.2.2  Superlative  Adjectives 

Superlative  adjectives,  denoted  by  [Adjective  -fsup],  are  formed  by  using  the  word  most 
before  an  adjective,  or  by  adding  the  -est  suffix.  Adjectives  generally  express  gradable 
attributes,  and  superlative  adjectives  express  the  very  top  end  of  the  scale  associated  with 
that  attribute. 


Pattern: 

List 

[Det] 

[Adjective  +sup] 

[Noun]. 

Example: 

List 

the 

highest 

salary. 

Pattern: 

[WH-Word] 

[BE-Verb] 

[Det] 

[Adjective  fsup] 

[Noun]  ? 

Example: 

Who 

is 

the 

newest 

employee? 

Pattern: 

[Det] 

[Adjective  -fsup] 

[Noun] 

[VP]. 

Example: 

The 

most  liberal 

cabinet  member 

was  killed  by  terrorists. 

Patterns: 

[NP] 

[Verb] 

[Det] 

[Adjective  fsup] 

[Noun]. 

Examples: 

Terrorists 

kidnapped 

the 

newest 

president. 

Guerrillas 

destroyed 

the 

newest 

bank  in  Anduhua. 

Criteria:  Demonstrated  understanding  by  correctly  interpreting  the  superlative  adjective 
as  qualifying  the  noun  in  accordance  with  the  attribute  expressed  by  the  adjective.  The 
attribute  has  a  scale  associated  with  it,  and  the  entity  expressed  by  the  noun  is  understood 
to  be  at  the  top  end  of  that  scale. 
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Score: 


4.3  Precentral  Premodifiers 

The  precentral  premodifiers  include  adverbs  which  qualify  the  gradable  adjective  which  func¬ 
tions  as  the  central  premodifier.  These  adverbs  further  specify  a  location  on  the  scale  asso¬ 
ciated  with  the  gradable  adjective.  For  example,  if  an  adjective  expresses  a  property,  then 
an  intensifier  ( “very”)  intensifies  or  raises  the  amount  of  that  property  ( “very  hungry”). 
Down-toners  (  “ little ,  ”)  lower  the  amount  of  that  property  (  “a  little  hungry”). 

Pattern:  List  [Det]  [Precentral]  [Central]  [Noun]. 

Example.  List  the  very  bigh  salaries. 

Pattern:  [WH-Word]  [BE- Verb]  [Det]  [Precentral]  [Central]  [Noun]  ? 

Example:  Who  is  the  very  newest  employee? 

Pattern:  [Det]  [Precentral]  [Central]  [Noun]  [VP]  . 

Example:  The  very  liberal  president  was  kidnapped  by  terrorists. 

Pattern:  [NP]  [Verb]  [Det]  [Precentral]  [Central]  [Noun] 

Examples:  Terrorists  kidnapped  the  brand  new  president. 

Guerrillas  burned  the  very  newest  facilities. 

Criteria:  Demonstrated  understanding  by  interpreting  the  [Precentral]  as  qualifying  the 
[Central]  which  expresses  a  gradable  attribute  of  the  noun. 

— - — - - — — - - — — — - — - - - Score: _ _ 

4.4  Postcentral  Premodifiers 

The  postcentral  premodifiers  include  passive  participles  (e.g.,  “hired”,  “paid”),  progressive 
participles  (e.g.,  “talking”,  “working”),  color  adjectives  (e.g.,  “yellow”),  and  others.  They 
are  called  “postcentral”  because  they  are  most  likely  to  occur  after  the  gradable  adjectives 
which  function  as  central  premodifiers  and  right  before  the  noun  head. 

In  the  examples  in  the  following  subsections,  the  postcentral  premodifiers  are  bracketed  for 
easy  identification. 

4.4.1  Passive  Participle  as  Premodifier 

The  passive  participle ,  denoted  by  [Participle  (-ed)],  is  a  form  of  a  verb  with  the  “-ed” 
or  -en  ending,  as  in  “hired”,  “arrested” ,  or  “spoken”,  or  certain  irregular  verbs  such  as 
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Pattern: 

List 

[Det] 

[Participle  (-ed)] 

[Noun]. 

Example: 

List 

the 

married 

employees  . 

Pattern: 

[WH-Word]  [BE-Verb] 

[Det] 

[Participle  (-ed)] 

[Noun]  ? 

Example: 

Who 

are 

the 

commissioned 

employees? 

Pattern: 

([Det]) 

[Participle  (-ed)] 

[Noun] 

(VP]. 

Examples: 

The 

elected 

official 

was  killed  by  terrorists. 

Armed 

men 

attacked  the  farm. 

Pattern: 

[NP] 

[Verb]  [Det] 

[Participle  (-ed)]  [Noun]. 

Example: 

Terrorists 

kidnapped  the 

worried 

president. 

Criteria:  Demonstrated  understanding  by  interpreting  the  [Participle  (-ed)]  as  expressing  a 
defining  attribute  of  the  entity  expressed  by  the  noun  which  it  premodifies. 

- — — . — - - Score: _ 


4.4.2  Progressive  Participle  as  Premodifier 

The  progressive  participle ,  denoted  [Participle  (-ing)],  is  a  form  of  a  verb  with  an  “-mg” 
ending. 


Pattern: 

List 

[Det] 

[Participle  (-ing)] 

[Noun]. 

Example: 

List 

the 

working 

employees 

Pattern: 

[WH-Word] 

[BE-Verb] 

[Det] 

[Participle  (-ing)] 

[Noun]  ? 

Example: 

Who 

is 

the 

hiring 

manager? 

Pattern: 

[Det] 

[Participle  (-ing)] 

[Noun] 

[VP]. 

Example: 

The 

kidnapping 

terrorists 

killed  the  mayor. 

Pattern: 

[NP] 

[Verb] 

[Det] 

[Participle  (-ing)] 

[Noun]. 

Examples: 

Terrorists 

kidnapped 

the 

working 

president. 

Guerrillas 

attacked 

a 

passing 

civilian. 

Criteria:  Demonstrated  understanding  by  interpreting  the  [Participle  as  expressing  a 

defining  attribute  of  the  entity  expressed  by  the  noun  which  it  premodifies. 

— - - - - - - - - - .Score: _ 


4.4.3  Infinitive  as  Premodifier  (t.b.d.) 
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4.5  Noun-Noun  Phrases  or  Nominal  Compounds 

A  nominal  compound  is  a  noun  phrase  that  consists  of  a  sequence  of  nouns.  In  the  current 
version  of  this  Procedure,  we  restrict  the  nominal  compounds  to  two  nouns.  Nominal  com¬ 
pounds  can  be  semantically  categorized  according  to  the  relationship  that  exists  between  the 
two  nouns  of  the  compound.  The  following  subsections  list  of  some  of  the  primary  recognized 
relationships  that  can  exist  between  the  two  nouns  of  a  nominal  compound.  In  this  section, 
[Noun]i  and  [Noun]2  represent  the  first  and  second  nouns  of  the  compound,  respectively. 

The  type  of  noun  phrases  we  will  test  in  this  section  has  the  pattern: 

([Det])  [Noun]i  [Noun]2  ([PP]) 

Using  any  of  the  basic  sentence  patterns,  replace  the  noun  phrase  with  this  nominal  com¬ 
pound  noun  phrase.  Noun-noun  phrases  will  be  bracketed  in  all  sentence  examples  for  ease 
of  identification. 

4.5.1  AT-TIME  Nominal  Compound  Type 

The  AT-TIME  nominal  compound  type  refers  to  a  noun-noun  phrase  in  which  the  object 
expressed  by  the  second  noun  occurs  at  or  during  the  time  expressed  by  the  first  noun.  We 
summarize  this  briefly  as: 

[Nounji  is  the  time  at  which  [Noun]2  occurs  OR 
[Noun]2  occurs  at  time  [Nounji 

An  example  of  such  a  nominal  compound  is  “ April  showers”  since  the  showers  occur  during 
April. 

Noun-noun  Phrase  Examples: 

Eg,  December  absences,  October  wages,  March  birthdate 
Eg,  April  showers,  winter  storm,  December  attack 

Full  Sentence  Examples: 

Eg,  Does  John  Smith  have  a  [December  birthdate]? 

Eg,  Two  civilians  were  killed  in  the  [January  10  attack]  on  Santa  Elena. 
Eg,  The  [December  bombing]  of  the  First  National  Bank  was  carried  out 
by  the  FMLN. 

Eg,  URNG  guerrillas  carried  out  the  [February  2  attack]  on  the  Santo 
Tomas  farm . 

Criteria:  Demonstrated  understanding  that  the  first  noun  expresses  the  time  at/during  which 
the  second  noun  occurs. 
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Score: 


4.5.2  AT-LOCATION  Nominal  Compound  Type 

The  AT-LOCATION  nominal  compound  type  refers  to  a  noun-noun  phrase  in  which  the 
object  expressed  by  the  second  noun  is  located  or  used  at  the  location  expressed  by  the  first 
noun.  We  summarize  this  briefly  as: 

[Nounji  is  the  place  at/in  which  [Noun]2  is  located  (or  used)  OR 
[Noun]2  is  (used)  at  location  [Nounji 

An  example  of  such  a  nominal  compound  is  “kitchen  table” since  this  type  of  table  is  (intended 
to  be)  used  in  a  kitchen. 

Noun-noun  Phrase  Examples: 

Eg,  kitchen  table,  shop  vacuum,  camp  stove 

Eg,  New  York  office,  Usulutan  village,  Andahua  bank 

Full  Sentence  Examples: 

Eg,  Does  John  Doe  work  in  the  [New  York  branch]? 

Eg,  Three  people  were  killed  in  an  explosion  at  an  [Andahua  bank] . 
Eg,  Three  civilians  were  injured  in  a  terrorist  attack  on  an 
[Usulutan  village] . 

Eg,  The  guerrillas  bombed  a  [Guatemala  City  bank] . 

Criteria:  Demonstrated  understanding  that  the  first  noun  expresses  the  place  where  the 
second  noun  is  located  or  used. 


Score: 


4.5.3  PURPOSE-CONCERNS  Nominal  Compound  Type 

The  PURPOSE-CONCERNS  nominal  compound  type  refers  to  a  noun-noun  phrase  in  which 
the  purpose  expressed  by  the  second  noun  concerns  the  object  expressed  by  the  first  noun. 
We  summarize  this  briefly  as: 

[Nounji  names  the  item  about  which  [Noun]2  is  concerned  OR 
The  purpose  of  [Noun]2  concerns  [Nounji 

An  example  of  such  a  nominal  compound  is  “milk  bottle”  since  the  purpose  of  the  bottle 
concerns  milk,  specifically  as  a  container  for  milk. 

Noun-noun  Phrase  Examples: 
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Eg,  bread  knife,  milk  bottle,  income  tax,  flea  collar 
Eg,  water  pipe,  house  paint,  pepper  mill,  oil  truck,  ski  mask 
Eg,  tax  law,  price  war,  product  brochure,  linguistics  book 
Eg,  war  bulletin,  news  bulletin 

Full  Sentence  Examples: 

Eg,  A  [news  bulletin]  announced  that  the  mayor  vas  kidnapped  yesterday. 
Eg,  In  a  [news  bulletin] ,  the  government  reported  the  guerrilla  attack 
on  the  embassy. 

Criteria:  Demonstrated  understanding  that  the  object  expressed  by  the  second  noun  is  used 
for  a  purpose  which  is  concerned  with  the  object  expressed  by  the  first  noun. 

_ Score: _ 


4.5.4  BELONGS-TO  Nominal  Compound  Type 

The  BELONGS-TO  nominal  compound  type  refers  to  a  noun-noun  phrase  in  which  the  object 
expressed  by  the  first  noun  has  possession  of,  or  responsibility  for,  the  object  expressed  by 
the  second  noun.  We  summarize  this  briefly  as: 

[Noun]2  belongs  to  [Nounji 

An  example  of  such  a  nominal  compound  is  “rebel  positions”  since  the  positions  belong  to 
the  rebels. 

Noun-noun  Phrase  Examples: 

Eg,  department  computer 

Eg,  rebel  positions,  rebel  casualties,  government  spokesman,  FAL  rifle 
Full  Sentence  Examples: 

Eg,  A  [government  spokesman]  announced  that  terrorists  kidnapped  the 
mayor . 

Eg,  A  [URNG  spokesman]  claimed  responsibility  for  the  attack  on  the 
Santo  Tomas  farm. 

Criteria:  Demonstrated  understanding  that  the  object  expressed  by  the  second  noun  belongs 
to  the  object  expressed  by  the  first  noun  . 

_ Score:  _ 


4.5.5  PART-OF  Nominal  Compound  Type 
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The  PART-OF  nominal  compound  type  refers  to  a  noun-noun  phrase  in  which  the  object 
expressed  by  the  second  noun  is  an  intrinsic  part  of  the  object  expressed  by  the  first  noun. 
We  summarize  this  briefly  as: 

[Noun]!  is  an  object  of  which  [Noun]2  is  a  part  OR 
[Noun]2  is  part  of  [Noun]i 

An  example  of  such  a  nominal  compound  is  “ car  wheel ”  since  the  wheel  is  (intended  to  be) 
part  of  a  car. 

Noun-noun  Phrase  Examples: 

Eg,  department  head,  car  wheel,  church  bell,  garage  door 
Eg,  National  Guard  units,  FMLN  troops,  patrol  leader 

Full  Sentence  Examples: 

Eg,  Is  John  Smith  a  [department  head]  ? 

Eg,  [FMLN  troops]  kidnapped  the  mayor  on  June  12. 

Criteria:  Demonstrated  understanding  that  the  object  expressed  by  the  second  noun  is  an 
intrinsic  part  of  the  object  expressed  by  the  first  noun. 

_ Score: _ 


4.5.6  PRODUCES  Nominal  Compound  Type 

The  PRODUCES  nominal  compound  type  refers  to  a  noun-noun  phrase  in  which  the  object 
expressed  by  the  first  noun  is  produced  by  the  object  expressed  by  the  second  noun.  We 
summarize  this  briefly  as: 

[Noun]i  is  produced  by  [Noun]2  OR 
[Noun]2  produces  [Nounji 

An  example  of  such  a  nominal  compound  is  “ coffee  machine ”  since  this  type  of  machine 
produces  (the  drink)  coffee. 

Noun-noun  Phrase  Examples: 

Eg,  coffee  machine,  tear  gas,  oil  well,  coffee  region 
Full  Sentence  Examples: 

Eg,  Terrorists  attacked  farmers  in  the  [coffee  region]  of  Peru. 

Eg,  The  guerrillas  attacked  the  [oil  wells] . 
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Criteria:  Demonstrated  understanding  that  the  object  expressed  by  the  first  noun  is  pro¬ 
duced  by  the  entity  expressed  by  the  second  noun. 

_ Score:  _ 

4.5.7  EXECUTED-BY  Nominal  Compound  Type 

The  EXECUTED-BY  nominal  compound  type  refers  to  a  noun-noun  phrase  in  which  the 
action  expressed  by  the  second  noun  was  executed  or  carried  out  by  the  object  expressed  by 
the  first  noun.  We  summarize  this  briefly  as: 

[Noun]j  executed  the  action  or  activity  [Nounjx  OR 
[Noun]2  was  executed  by  Noun]i 

An  example  of  such  a  nominal  compound  is  “terrorist  attack”  since  the  attack  was  carried 
out  by  the  terrorists. 

Noun-noun  Phrase  Examples: 

Eg,  terrorist  attack,  FMLN  bombing 
Full  Sentence  Examples: 

Eg,  The  mayor  was  killed  in  the  [terrorist  attack] . 

Eg,  A  civilian  was  killed  in  the  [guerrilla  attack] . 

Eg,  The  [FMLN  bombing]  of  the  bank;  injured  three  civilians. 

Criteria:  Demonstrated  understanding  that  the  action  expressed  by  the  second  noun  was 
executed  or  carried  out  by  the  object  expressed  by  the  first  noun. 

_ Score: _ 


4.5.8  MADE-OF  Nominal  Compound  Type 

In  this  and  the  following  subsections,  we  provide  only  the  succinct  summary  of  the  relation 
and  examples. 

[Noun]i  is  a  material  from  which  [Nounjj  is  made  OR 
[Noun]2  is  made  of  material  [Noun]i 

Eg,  glass  bottle,  cardboard  box,  bran  muffins 
Eg,  dynamite  sticks,  wood  houses 


Eg,  The  URNG  guerrillas  threw  [dynamite  sticks]  at  the  embassy 
Eg.  Guerrillas  burned  the  [wood  houses]  in  Flores. 
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Criteria:  Demonstrated  understanding  that  the  object  expressed  by  the  first  noun  is  a  sub¬ 
stance  of  which  the  object  expressed  by  the  second  noun  is  made. 

_ Score: _ 


4.5.9  PRODUCED-BY  Nominal  Compound  Type 

[Nounji  produces  [Noun)2  OR 
[Noun]2  is  produced  by  [Nounji 

Eg,  artillery  fire,  bomb  blast 

Eg,  Three  civilians  were  injured  by  [artillery  fire]  in  a  terrorist 
attack  on  Andahua. 

Eg,  (Preliminary  Sentence:  Guerrillas  planted  a  bomb  at  the  Central 
Bank.)  The  [bomb  blast]  killed  the  bank  manager. 

Criteria:  Demonstrated  understanding  that  the  object  expressed  by  the  second  noun  is 
produced  by  the  object  expressed  by  the  first  noun. 

_ Score: _ 


4.5.10  PURPOSE-BENEFITS  Nominal  Compound  Type 

[Nounji  is  a  beneficiary  of  [Noun]2  OR 
The  purpose  of  [Noun]2  benefits  [Nounji 

Eg,  employee  benefits,  baby  food 

Eg,  What  [employee  benefits]  does  John  Smith  have? 

Eg,  The  guerrillas  stole  [baby  food]  from  the  clinic. 

Criteria:  Demonstrated  understanding  that  the  object  or  entity  expressed  by  the  first  noun 
benefits  from  the  object  expressed  by  the  second  noun. 

_ Score: _ 


4.5.11  PURPOSE-IS  Nominal  Compound  Type 

[Noun]x  names  the  purpose  of  [Noun]2  OR 
The  purpose  of  [Noun]2  is  [Nounji 

Eg,  delivery  truck,  bombing  raid 

Eg,  Guerrillas  conducted  a  [bombing  raid]  on  the  Santo  Tomas  farm. 
Eg,  Guerrillas  robbed  the  [delivery  truck] . 
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Criteria:  Demonstrated  understanding  that  the  object  or  entity  expressed  by  the  second 
noun  is  for  the  purpose  of  that  which  is  expressed  by  the  first  noun. 

_ Score: _ 


4.5.12  HAS-TYPE  Nominal  Compound  Type 

[NounJj  names  the  type  of  [Noun]2  OR 
[Noun]2  is  of  type  [Noun]i 

Eg,  laser  printer,  apple  tree 

Criteria:  Demonstrated  understanding  that  the  object  or  entity  expressed  by  the  complete 
noun  phrase  is  a  subtype  of  the  type  of  object  expressed  by  the  second  noun.  The  subtype 
specification  is  given  by  the  first  noun. 

_ Score: _ 


4.5.13  USES  Nominal  Compound  Type 


[Noun]x  is  used  by  [Noun]2  OR 

The  object  [Noun]2  uses  the  object  [Noun]x 

Eg,  steam  iron,  oil  furnace,  gasoline  generators 

Eg,  The  guerrilla  attack  on  the  farm  destroyed  the  [gasoline 
generators] . 


Criteria:  Demonstrated  understanding  that  the  object  expressed  by  the  second  noun  uses 
the  object  expressed  by  the  first  noun. 

_ Score: _ 


4.5.14  HAS-PART  Nominal  Compound  Type 

[Nounjx  is  a  part  of  [Noun]2  OR 
[Noun]2  has  [Nounji  as  a  part 

Eg,  goose-neck  lamp,  roll-top  desk 
Eg,  tank  truck,  double-barrel  shotgun 


Eg,  The  guerrillas  burned  the  [tank  truck] . 

Eg,  The  guerrilla  assassinated  the  President  with  a  [double-barrel 
shotgun] . 
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Criteria:  Demonstrated  understanding  that  the  object  express  t  by  the  second  noun  has  the 
object  expressed  by  the  first  noun  as  a  part. 

_ _ Score: _ 

4.6  Premodifier  Combinations 

In  this  subsection,  we  will  test  combinations  of  the  noun  phrase  premodifiers  covered  in  the 
several  preceding  sections. 

(a)  [[Det]  [Central]  [Central]  [Noun]] 

Pattern:  List  [[Det]  [Central]  [Central]  [Noun]]. 

Example:  List  the  new  young  employees. 

Pattern:  [WH-Word]  [Verb]  [[Det]  [Central]  [Central]  [Noun]]. 

Pattern:  [[Det]  [Central]  [Central]  [Noun]]  [VP]. 

Example:  The  expensive  new  Government  House  was  attacked. 

Pattern:  [NP]  [Verb]  [[Det]  [Central]  [Central]  [Noun]]. 

Example:  The  terrorists  attacked  the  expensive  new  Government  house. 

Criteria:  Demonstrated  understanding  of  the  attributes  indicated  of  the  noun  phrase  by  the 
central  premodifier  combination. 

- - - Score: _ 

(b)  [[Det]  [Central]  [Precentral]  [Central]  [Noun]] 

Pattern:  List  [[Det]  [Central]  [Precentral]  [Central]  [Noun]]. 

Example:  List  the  new  very  young  employees. 

Pattern:  [WH-Word]  [Verb]  [[Det]  [Central]  [Precentral]  [Central]  [Noun]]. 

Pattern:  [[Det]  [Central]  [Precentral]  [Central]  [Noun]]  [VPj. 

Example:  The  new  very  expensive  Government  House  was  attacked. 

Pattern:  [NP]  [Verb]  [[Det]  [Central]  [Precentral]  [Central]  [Noun]]. 

Criteria:  Demonstrated  understanding  of  the  attributes  indicated  of  the  noun  phrase  by  the 
central  and  precentral  premodifier  combination. 

- - — - - Score: _ 

(c)  [[Det]  [Central]  [Participle]  [N  oun]] 
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Pattern:  List 

Example:  List 


[[Det]  [Central]  [Participle]  [N  oun]]. 
the  current  hiring  manager. 


Pattern:  [WH-Word]  [Verb]  [[Det]  [Central]  [Participle]  [Noun]]. 

Pattern:  [[Det]  [Central]  [Participle]  [Noun]]  [VP]. 

Example:  The  Marxist  kidnapping  terrorists  demanded  a  high  ransom. 

Pattern:  [NP]  [Verb]  [[Det]  [Central]  [Participle]  [Noun]]. 

Example:  The  guerrillas  conducted  a  quick  bombing  raid. 

Criteria:  Demonstrated  understanding  of  the  attributes  indicated  of  the  noun  phrase  by  the 
central  and  participle  premodifier  combination. 

_ _ Score: _ 


(d)  [[Det]  [Precentral]  [Central]  [Participle]  [Noun]] 

Pattern:  List  [[Det]  [Precentral]  [Central]  [Participle]  [Noun]]. 

Example:  List  the  brand  new  hiring  manager. 

Pattern:  [WH-Word]  [Verb]  [[Det]  [Precentral]  [Central]  [Participle]  [Noun]]. 

Pattern:  [[Det]  [Precentral]  [Central]  [Participle]  [Noun]]  [VP]. 

Examplel:  The  very  new  working  Prime  Minister  was  kidnapped 

by  terrorists. 

Example2:  A  very  quick  bombing  raid  killed  a  civilian. 

Pattern:  [NP]  [Verb]  [[Det]  [Precentral]  [Central]  [Participle]  [Noun]]. 

Criteria:  Demonstrated  understanding  of  the  attributes  indicated  of  the  noun  phrase  by  the 
precentral,  central,  and  participle  premodifier  combination. 

_ Score: _ 


(e)  [[Det]  [Participle]  [Participle]  [Noun]] 

Pattern:  List  [[Det]  [Participle]  [Participle]  [Noun]]. 

Example:  List  the  retired  supervising  manager. 

Pattern:  [WH-Word]  [Verb]  [[Det]  [Participle]  [Participle]  [Noun]]. 

Pattern:  [[Det]  [Participle]  [Participle]  [Noun]]  [VP]. 

Examples:  The  frightened  kidnapping  terrorists  demanded  a  high  ransom. 

The  injured  kidnapped  peasants  escaped  from  the  guerrillas. 

Pattern:  [NP]  [Verb]  [[Det]  [Participle]  [Participle]  [Noun]]. 

Criteria:  Demonstrated  understanding  of  the  attributes  indicated  of  the  noun  phrase  by  the 
participle  premodifier  combination. 
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Score: 


(f)  [[Det]  [Central]  [Participle]  [Participle]  [Noun]] 

Pattern:  List  [[Det]  [Central]  [Participle]  [Participle]  [Noun]]. 

Example:  List  the  young  newly  hired  married  employees. 

Pattern:  [WH-Word]  [Verb]  [[Det]  [Central]  [Participle]  [Participle]  [Nounl]. 

Pattern:  [[Det]  [Central]  [Participle]  [Participle]  [Noun]]  [VP]. 

Example:  The  young  frightened  kidnapping  terrorists  demanded  a  high  ransom. 

The  young  injured  kidnapped  peasants  escaped  from  the  guerrillas. 


Pattern:  [NP]  [Verb]  [[Det]  [Centre!]  [Participle]  [Participle]  [Noun]]. 

Criteria:  Demonstrated  understanding  of  the  attributes  indicated  of  the  noun  phrase  by  the 
central  and  participles  premodifier  combination. 

_ Score: _ 


(g)  [[Det]  [Precentral]  [Participle]  [Noun]] 

Pattern:  List  [[Det]  [Precentral]  [Participle]  [Noun]]. 

Example:  List  the  most  experienced  manager. 

Pattern:  [WH-Word]  [Verb]  [[Det]  [Precentral]  [Participle]  [Noun]]. 

Pattern:  [[Det]  [Precentral]  [Participle]  [Noun]]  [VP]. 

Examples:  The  newly  appointed  President  was  kidnapped  by  terrorists. 

The  badly  injured  peasants  escaped  from  the  guerrillas. 

Pattern:  [NP]  [Verb]  [[Det]  [Precentral]  [Participle]  [Noun]]. 

Criteria:  Demonstrated  understanding  of  the  attributes  indicated  of  the  noun  phrase  by  the 
participle  premodifier  combination. 

_ Score: _ 


5.  Genitives  and  Alternative  Forms 
5.1  Genitives 

The  genitive  case  for  regular  nouns  is  indicated  with  an  apostrophe  followed  by  the  letter 
“s”  (called  “ apostrophe  s”)  for  singular  nouns  not  ending  in  “s”  (  “boy’s”),  and  with  a  final 
apostrophe  for  plural  nouns  ending  in  “s”  (“boys’”).  For  irregular  plural  nouns  which  do 
not  end  in  s,  the  apostrophe  s  is  also  used.  For  example,  for  the  irregular  noun  “ child ”,  the 
singular  genitive  is  “child’s”  and  the  plural  genitive  is  “children’s”. 
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With  singular  nouns  and  names  ending  in  the  phonetic  /z/  sound  (  “ Jones ”,  “Dickens”, 
“Socrates”)  the  genitive  inflection  is  written  as  either  a  final  apostrophe  or  apostrophe  s, 
with  a  preference  for  the  solo  apostrophe.  The  spoken  inflection  is  also  variable,  with  a 
preference  towards  the  extra  / iz/  sound  in  single  syllable  words  and  names,  and  no  extra 
sound  in  two  or  more  syllable  words.  With  singular  nouns  and  names  ending  in  the  phonetic 
/s/  sound  (  “Ross”,  “niece”)  the  genitive  inflection  is  apostrophe  s  and  the  extra  /iz/  sound 
is  pronounced. 


The  genitive  is  formed  from  a  noun  phrase  and  functions  as  either  a  central  determiner  or  a 
premodifier  in  a  larger,  superordinate  noun  phrase. 


As  a  centra*  determiner,  the  genitive  fills  a  slot  in  the  superordinate  noun  phrase  equivalent 
to  a  central  determiner  such  as  “the”,  and  may  follow  a  precentral  determiner  or  precede 
noun  phrase  modifiers: 


the 
Jenny’s 
(both)  the 
(both)  Jenny’s 


(new)  desk, 
(new)  desk, 
(new)  teachers, 
(new)  teachers. 


Both  the  noun  phrase  from  which  the  genitive  is  formed,  and  the  superordinate  noun  phrase 
of  which  it  is  the  determiner,  can  have  modifiers.  In  the  following  examples  the  genitive 
noun  phrase  is  enclosed  by  brackets  for  ease  of  identification. 


[Jo/m’s]  new  job 

[the  manager’s]  salary 

[the  previous  manager’s ]  retirement  date 

Note  that  a  genitive  case-marker  marks  a  complete  noun  phrase,  not  just  a  noun.  In  the 
following  example  the  genitive  case-marker  is  on  the  end  of  the  prepositional  phrase  “of 
underwriting”  which  is  a  post-modifier  of  the  noun  phrase  “the  director”,  although  it  is  the 
director  that  has  the  secretary.  Both  the  major  noun  phrase  and  its  premodifying  genitive 
noun  phrase  are  enclosed  with  brackets  for  ease  of  identification. 

[  [The  director  of  underwriting’s]  secretary]  is  ret' ring. 

The  following  example  indicates  that  there  are  what  might  be  considered  “performance” 
constraints  (what  a  language  user  is  able  to  understand)  on  how  large  a  genitive-marked 
noun  phrase  may  be. 

??[  [That  director  who  called  the  meeting  last  week’s]  secretary ]  is  retiring. 

NOTE:  “??”  signifies  that  the  sentence  which  it  marks  is  unacceptable  to  most  speakers. 

The  main  use  of  the  genitive  is  to  express  possession,  ranging  from  the  strict  sense  of  own¬ 
ership  (  “John’s  book”)  to  a  looser  sense  of  relationship  (  “Mary’s  husband”,  “Paul’s  doctor”). 

Other  uses  include,  but  are  not  limited  to: 
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•  the  subjective  genitive,  in  which  the  noun  phrase  being  premodified  expresses  an  event 
and  the  genitive  case-marked  noun  phrase  expresses  the  subject  of  that  event.  For 
example,  in: 


the  boy’s  application  to  college 
the  event  is  applying  to  college  and  the  subject  is  the  boy. 

•  the  objective  genitive,  in  which  the  noun  phrase  being  premodified  expresses  an  event 
and  the  genitive  case-marked  noun  phrase  expresses  the  object  of  that  event.  For 
example,  in: 

the  boy’s  release  from  jail 

the  event  is  a  release  from  jail  and  the  object  is  the  boy. 

•  the  genitive  of  origin,  in  which  the  noun  phrase  being  premodified  expresses  an 
or  entity  and  the  genitive  case-marked  noun  phrase  expresses  the  origin  of  that 
or  entity,  as  in: 

the  girl’s  story 
the  general’s  letter 

•  the  genitive  of  meusure ,  in  which  the  noun  phrase  being  premodified  expresses  an 
or  entity  and  the  genitive  case- marked  noun  phrase  expresses  a  measure  of  that 
or  entity,  as  in: 

a  ten  day’s  absence 

•  the  genitive  of  attribute,  in  which  the  noun  phrase  being  premodified  expresses  an 
attribute  of  the  object  or  entity  expressed  by  the  genitive  case-marked  noun  phrase, 
as  in: 


object 

object 


object 

object 


the  man’s  height 
John  Smith’s  birthday 

•  the  partitive  genitive ,  in  which  the  premodified  noun  phrase  expresses  an  intrinsic  part 
of  the  object  or  entity  expressed  by  the  genitive  case- marked  noun  phrase,  for  example, 
in: 


the  earth’s  crust 
the  baby’s  blue  eyes 

•  the  descriptive  genitive,  in  which  the  genitive  case-marked  noun  phrase  has  a  classifying 
role  with  respect  to  the  noun  phrase  being  premodified.  The  premodified  noun  phrase 
does  not  have  a  descriptive  relationship  of  any  sort  to  the  object  or  entity  expressed 
by  the  genitive  case-marked  noun  phrase.  For  example,  in  the  sentences: 

a  women’s  college 
a  ship’s  doctor 


III-22 


“ college ”  does  not  describe  “women”,  nor  does  “ doctor ”  describe  the  “ship”.  Rather, 
“women’s”  describes  the  kind  of  college,  and  “ship’s”  describes  the  kind  of  doctor.  A 
distinguishing  mark  of  this  type  of  genitive  is  the  fact  that  any  modifiers  and  deter¬ 
miners  preceding  it  generally  belong  to  the  head  noun,  rather  than  to  the  genitive 
noun. 

In  the  following  example,  there  are  three  interpretations  of  the  genitive,  one  in  which  John 
owns  the  paintings  (the  possessive  genitive),  one  in  which  John  is  the  painter  (the  subjective 
genitive ),  and  one  in  which  he  is  the  object  of  the  painting  (the  objective  genitive ): 

John’s  portraits  are  very  valuable  now. 

Which  interpretation  will  be  preferred  depends  a  great  deal  on  situational  or  other  pragmatic 
considerations.  However,  there  also  seems  to  be  a  hierarchical  organization  to  the  assignment 
of  semantic  roles,  in  which  possession  is  the  highest,  then  agency,  then  patient  or  theme  (i.e., 
the  animate  or  inanimate  object  upon  which  the  action  operates).  Thus,  in  the  sentence 

I  like  Mary’s  husband’s  painting, 

the  most  salient  interpretation  is  probably  that  which  has  Mary  as  the  possessor  of  her 
husband,  loosely  speaking,  and  her  husband  as  either  the  owner  or  the  painter.  The  less 
salient  reading  is  that  which  has  Mary  as  the  possessor  and  the  husband  as  the  subject  of 
the  painting.  On  the  other  hand,  in  a  sentence  like 

This  manager’s  annual  performance  evaluation  was  very  good, 

the  preference  is  not  nearly  as  obvious,  due  to  pragmatics,  and  in  fact  it  may  be  as  likely  for 
this  manager  to  be  interpreted  as  the  subject  of  the  evaluation  than  as  the  agent  performing 
the  evaluation.  Thus,  it  is  clear  that  both  predisposition  and  pragmatics  must  be  taken  into 
consideration  in  order  to  correctly  assign  meaning  to  this  type  of  construction. 

Possessives 

There  are  a  number  of  ways  of  expressing  possession  in  English,  the  three  most  important 
being  1)  use  of  the  genitive  case,  2)  use  of  a  preposition,  3)  use  of  an  explicit  verb  of 
possession.  The  three  cases  are  illustrated  by  the  examples  below: 

1)  John’s  book 
Whose  book 

2)  The  children  of  the  senator 

The  senator  with  the  attractive  wife 

3)  I  have  a  book 
I  own  a  book 
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The  system  for  expressing  possession  is  very  rich  in  English  because  of  the  possibility  of 
using  not  only  these  three  basic  syntactic  means  for  indicating  possession,  but  also  because 
of  the  possibility  of  having  combinations  of  the  three.  Thus,  possessives  can  be  expressed  by 
using  both  genitive  case  and  the  preposition  of. 

That  book  of  John’s. 

both  the  genitive  and  an  explicit  verb  of  possession: 

John  has  Harry’s  book. 
both  a  preposition  and  a  verb: 

The  senator  has  the  children  of  his  first  wife. 

or  all  three: 

I  still  have  that  record  of  John’s  that  I  borrowed  last  week. 

In  addition  to  all  of  this,  we  have  the  possibility,  in  principle,  of  stringing  together  an 
unlimited  number  of  genitive  case-marked  nouns: 

John’s  book 

John’s  supervisor’s  book 

John’s  supervisor’s  associate’s  book 

In  this  section  on  noun  phrases,  we  will  not  test  possessives  which  are  expressed  using  a 
verb. 


5.1.1  Genitive  Constructions 

5. 1.1.1  Genitive  with  Simple  Noun  Phrase 

Test  a  simple  noun  phrase,  that  is,  one  without  premodifiers  or  postmodifiers,  with  the 
genitive  case  marking. 

Pattern:  List  [NP  -{-genitive]  [NP]. 

Example:  List  John  Smith’s  address. 

Pattern:  [WH-Word]  ([NP])  [BE- Verb]  [NP  -{-genitive]  [NP]  ? 

Example:  What  is  Smith’s  salary  ? 

Pattern:  [NP]  [Verb]  [NP  -{-genitive]  [NP]. 

Examples:  Terrorists  kidnapped  the  Governor’s  son. 

Guerrillas  kidnapped  the  Mayor’s  wife. 
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Pattern:  [NP  -(-genitive]  [NP]  [VP]. 

Example:  The  Governor’s  son  was  kidnapped  by  terrorists. 

Criteria:  Demonstrated  understanding  of  the  relationship  between  the  entities  expressed  by 
the  noun  phrase  with  the  genitive  case  marking  and  the  noun  phrase  being  premodified. 

_  Score:  _ 

5. 1.1. 2  Genitive  with  Complex  Noun  Phrase 

In  this  test,  use  a  complex  noun  phrase  (i.e.,  one  which  includes  premodification  or  post¬ 
modification)  as  the  noun  phrase  with  the  genitive  case  marking. 

Pattern:  List  [NP  -(-genitive]  [NP]. 

Examples:  List  the  largest  department’s  members. 

List  the  Chairman  of  the  Board’s  members. 

Pattern:  [WH-Word]  ([NP])  [BE- Verb]  [NP  +genitive]  [NP]  ? 

Example:  What  is  the  newest  employee’s  name  ? 

Pattern:  [NP]  [Verb]  [NP  -f genitive]  [NP]. 

Examples:  Terrorists  kidnapped  the  Mayor  of  Cuilapa’s  son. 

Guerrillas  kidnapped  the  Mayor  of  Flore’s  wife. 

Pattern:  [NP  -f genitive]  [NP]  [VP]. 

Example:  The  Mayor  of  Cuilapa’s  son  was  kidnapped  by  terrorists. 

Criteria:  Demonstrated  understanding  of  the  relationship  between  the  entities  expressed  by 
the  noun  phrase  with  the  genitive  case  marking  and  the  noun  phrase  being  premodified. 

_ Score: _ 

5. 1.1. 3  Multiple  Genitives 

Multiple  genitives  have  the  following  pattern: 

Pattern:  [NP  +genitive]i  [NP  -|-genitive]2  [NP]3 

Example:  the  Chairman’s  secretary’s  salary 

where  there  is  a  genitive  relationship  between  [NP]2  and  [NP]3  and  between  [NP]j  and  [NP]2. 

Pattern:  List  [[NP  -(-genitive]  [NP  -(-genitive]]  [NP]]. 

Examples:  List  the  company’s  department’s  names  . 

List  the  Chairman  of  the  Board’s  secretary’s  salary? 

Pattern:  [WH-Word]  ([NP])  [BE- Verb]  [[[NP  -(-genitive]  [NP  -(-genttive]]  [NP]]  ? 

Example:  What  is  John  Smith’s  supervisor’s  name? 

Pattern:  [NP]  [Verb]  [[NP  -(-genitive]  [NP  -(-genitive]]  [NP]]. 

Example:  Terrorists  kidnapped  the  Mayor  of  Cuilapa’s  son’s  wife. 
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Pattern:  [[NP  -(-genitive]  [NP  -(-genitive]]  [NF]]  [VP], 

Examples:  Cuilapa’s  Mayor’s  sod  was  kidnapped  by  terrorists. 

The  Mayor  of  Flore’s  son’s  wife  was  kidnapped  by  guerrillas. 

Criteria:  Demonstrated  understanding  of  the  relationship  between  the  first  [NP  -(-genitive] 
and  the  second,  and  the  relationship  between  the  second  [NP  -(-genitive]  and  the  noun  phrase 
following. 

_ Score: _ 

5.1.2  Semantics  of  Genitives 

5. 1.2.1  Possessive  Genitive 

The  possessive  genitive  expresses  a  number  of  related  concepts,  ranging  from  the  strict 
sense  of  ownership  (“John’s  book”)  to  a  looser  sense  of  relationship  (“Mary’s  husband”, 
“Paul’s  doctor”),  with  the  genitive  case-marked  noun  phrase  expressing  the  “owner”  and 
the  premodified  noun  phrase  expressing  the  “possession”. 

Pattern:  List  [NP  -f genitive]  [NP]. 

Example:  List  John  Smith’s  salary. 

Pattern:  [WH-Word]  [BE- Verb]  [NP  -(-genitive]  [NP]  ? 

Example:  Who  are  Smith’s  past  employers  ? 

Pattern:  [NP]  [Verb]  [NP  -(-genitive]  [NP]. 

Example:  Terrorists  bombed  the  Government’s  offices. 

Pattern:  [NP  -(-genitive]  [NP]  [VP]. 

Example:  The  Governor’s  son  was  kidnapped  by  terrorists. 

Criteria:  Demonstrated  understanding  of  the  possessive  relationship  (in  the  loose  sense  of 
“possession”)  between  th**  entities  expressed  by  the  noun  phrase  with  the  genitive  case 
marking  and  the  noun  phrase  being  premodified. 

_ _  Score: _ 

5. 1.2.2  Subjective  Genitive 

In  the  subjective  genitive ,  the  noun  phrase  being  premodified  expresses  an  event  and  the 
genitive  case-marked  noun  phrase  expresses  the  subject  of  that  event.  For  example,  in: 

the  boy’s  application  to  college 

the  event  is  applying  to  college  and  the  subject  is  the  boy. 

Pattern:  List  [NP  -(-genitive]  [NP]. 

Example:  List  John  Smith’s  sales. 

Pattern:  [WH-Word]  [BE- Verb]  [NP  -(-genitive]  [NP]  ? 

Example:  What  were  the  Chicago  branch’s  sales? 
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Pattern:  [NP]  [Verb]  [NP  -f genitive]  [NP]. 

Example:  ARENA  sponsored  the  terrorists’  attacks. 

Pattern:  [NP  +genitive]  [NP]  [VP]. 

Example:  The  terrorists’  attacks  destroyed  the  Government  House. 

Criteria:  Demonstrated  understanding  of  the  subject-event  relationship  between  the  entities 
expressed  by  the  noun  phrase  with  the  genitive  case  marking  and  the  noun  phrase  being 
premodified. 

_ Score: _ 


5. 1.2. 3  Objective  Genitive 

In  the  objective  genitive ,  the  noun  phrase  being  premodified  expresses  an  event  and  the 
genitive  case-marked  noun  phrase  expresses  the  object  of  that  event.  For  example,  in: 

the  boy’s  release  from  jail 


the  event  is  a  release  from  jail  and  the  object  is  the  boy. 

Pattern:  List  [NP  +genitive]  [NP]. 

Example:  List  John  Smith’s  hiring. 

Pattern:  [WH-Word]  [BE- Verb]  [NP  +genitive]  [NP]  ? 

Example:  What  was  John  Smith’s  promotion? 

Pattern:  [NP]  [Verb]  [NP  +genitive]  [NP]. 

Example:  ARENA  sponsored  the  Governor’s  assassination. 


Pattern:  [NP  +genitive]  [NP]  [VP]. 

Example:  The  Mayor’s  kidnapping  was  done  by  Shining  Path. 


Criteria:  Demonstrated  understanding  of  the  object-event  relationship  between  the  entities 
expressed  by  the  noun  phrase  with  the  genitive  case  marking  and  the  noun  phrase  being 
premodified. 


Score: 


5. 1.2.4  Attributive  Genitive 

In  the  genitive  of  attribute,  the  noun  phrase  being  premodified  expresses  an  attribute  of  the 
object  or  entity  expressed  by  the  genitive  case-marked  noun  phrase,  as  in: 

the  man’s  height 
John  Smith ’s  birthday 

Pattern:  List  [NP  +  genitive]  [NP]. 

Example:  List  John  Smith’s  hire  date. 
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Pattern:  [WH-Word]  [BE- Verb]  [NP  +oen«tive]  [NP]  ? 

Example:  What  is  Smith’s  education  ? 

Pattern:  [NP  -(-genitive]  [NP]  [VP]. 

Example:  The  government’s  size  was  diminished  by  repeated  kidnappings. 

Criteria:  Demonstrated  understanding  that  the  noun  phrase  being  premodified  expresses  an 
attribute  of  the  object  expressed  by  the  noun  phrase  with  the  genitive  case  marking. 

_  Score:  _ 

5. 1.2. 5  Partitive  Genitive 

In  the  partitive  genitive ,  the  premodified  noun  phrase  expresses  an  intrinsic  part  of  the  object 
or  entity  expressed  by  the  genitive  case- marked  noun  phrase,  for  example,  in: 

the  earth’s  crust 
the  baby’s  blue  eyes 

Pattern:  List  [NP  +genitive]  [NP]. 

Example:  List  the  company’s  branches. 

Pattern:  [WH-Word]  [BE- Verb]  [NP  -(-genitive]  [NP]  ? 

Example:  What  are  the  company’s  branches? 

Pattern:  [NP]  [Verb]  [NP  -(-genitive]  [NP]. 

Example:  Terrorists  exploded  the  Government  House’s  roof. 

Pattern:  [NP  +genitive]  [NP]  [VP]. 

Example:  The  Mayor’s  arm  was  broken  by  terrorists. 

Criteria:  Demonstrated  understanding  of  the  part-whole  relationship  between  the  entities 
expressed  by  the  noun  phrase  with  the  genitive  case  marking  and  the  noun  phrase  being 
premodified. 

_ Score: _ 

5. 1.2.6  Descriptive  Genitive 

In  cases  where  a  genitive  is  a  descriptive  genitive,  it  is  used  as  a  modifier  of  a  noun  head,  and 
fills  a  premodification  slot  in  the  superordinate  noun  phrase.  Descriptive  genitives  have  a 
classifying  role  similar  to  that  of  noun  modifiers.  In  the  following  examples,  the  descriptive 
genitive  noun  phrase  and  the  superordinate  noun  phrase  into  which  it  is  embedded  are 
enclosed  by  brackets. 

There  are  [several  [women’s]  colleges ]  in  New  York. 

John  wants  to  become  [a  [s/up’s]  doctor\  when  he  graduates. 


A  distinguishing  mark  of  descriptive  genitives  is  the  fact  that  any  modifiers  and  determiners 


preceding  it  generally  belong  to  the  noun  head,  rather  than  to  the  genitive  noun.  For 
example, 

a  quaint  old  shepherd’s  cottage 


probably  comments  on  the  age  and  quaintness  of  the  cottage,  not  the  shepherd. 


Most  systems  will  not  be  able  to  discern  the  subtleties  of  the  descriptive  nature  of  descriptive 
genitives.  However,  a  system  should  hopefully  be  able  to  distinguish  which  noun  head  the 
determiners  and  premodifiers  belong  to.  In  order  to  test  descriptive  genitives,  make  sure 
there  is  premodification  or  predetermination  of  the  noun  head  in  addition  to  the  genitive 
case  marked  noun  phrase. 

Pattern:  List  ([Det])  ([Premodification])  [NP  -(-genitive]  [Head]. 

Example:  List  the  largest  employee’s  salary. 


Pattern:  [WH-Word]  [BE- Verb]  ([Det])  ([Premodification])  [NP  -(-genitive]  [Head]  ? 

Example:  What  is  the  largest  employee’s  salary  ? 


Pattern:  [NP]  [Verb]  ([Det])  ([Premodification])  [NP  -(-genitive]  [Head]. 

Example:  Terrorists  attacked  the  many  Government’s  offices. 

Pattern:  ([Det])  ([Premodification])  [NP  -(-genitive]  [Head]  [VP]. 

Example:  Well-trained  ARENA’S  guards  kidnapped  the 

Prime  Minister. 


Criteria:  Demonstrated  understanding  that  the  genitive  case-marked  noun  phrase,  and  the 
determiners  and  modifiers  preceding  it  all  modify  the  noun  head  following  the  genitive. 

_ Score: _ 


5.2  OF  as  an  Alternative  to  the  Genitive 

An  alternative  form  to  the  genitive  for  expressing  many  of  the  same  relationships  is  the 
prepositional  o/-phrase.  Below  are  some  examples  of  genitives  and  the  corresponding  of- 
phrase  forms: 


The  possessive  genitive:  “Mary’s  husband” 

Corresponding  o/-phrase:  the  husband  of  Mary 

The  subjective  genitive:  the  boy’s  application  to  college 
Corresponding  o/-phrase:  the  application  of  the  boy  to  college 

The  objective  genitive:  the  boy’s  release  from  jail 
Corresponding  o/-phrase:  the  release  of  the  boy  from  jail 


The  genitive  of  measure:  a  ten  day’s  absence 
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Corresponding  o/-phrase:  an  absence  often  days 

The  genitive  of  attribute:  John  Smith ’s  birthday 
Corresponding  o/-phrase:  the  birthday  of  John  Smith 

The  partitive  genitive:  the  earth ’s  crust 

Corresponding  o/-phrase:  the  crust  of  the  earth 

An  of- construction  has  the  pattern: 

Pattern:  [NP]j  of  [NP]2 

Example:  the  address  of  John  Smith 

Test  the  o/-phrase  as  an  alternative  form  to  the  genitive.  You  may  test  it  with  any  of  the 
above  semantic  relationships. 

Pattern:  List  [[NP]  of  [NP]]. 

Example:  List  the  address  of  John  Smith. 

Pattern:  [WH-Word]  ([NP])  [BE- Verb]  [[NP]  of  [NP]]  ? 

Example:  What  is  the  salary  of  John  Smith  ? 

Pattern:  [NP]  [Verb]  [[NP]  of  [NP]]. 

Example:  Terrorists  kidnapped  the  son  of  the  Governor. 

Pattern:  [[NP]  of  [NP]]  [VP]. 

Example:  The  son  of  the  Governor  was  kidnapped  by  terrorists. 

Criteria:  Demonstrated  understanding  of  the  relationship  between  the  noun  phrase  being 
postmodified  and  the  postmodifying  o/-phrase. 

_ Score: _ 

5.3  WITH  as  an  Alternative  to  the  Genitive 

An  alternative  form  to  the  genitive  for  expressing  possession  is  the  prepositional  with-phrase. 
The  uttt/i-construction  has  the  pattern: 

Pattern:  [NP]i  with  [NP]2 

Example:  the  woman  with  the  child 

where  [NP]!  is  postmodified  by  [NP]2. 

Test  the  with-phrase  as  an  alternative  form  to  the  genitive.  You  may  test  it  with  any  of  the 
above  semantic  relationships. 

Pattern:  List  [[NP]  with  [NP]]. 

Example:  List  the  employees  with  PhDs. 
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Pattern: 

Example: 

Pattern: 

Example: 


[WH-Word]  ([NP])  [BE- Verb]  [[NP]  with  [NP]]  ? 

What  is  the  department  with  the  highest  salaries. 


[NP]  [Verb]  [[NP]  with  [NP]]. 

Terrorists  kidnapped  the  Governor  with  Marxist  ideals. 


Example:  (Preliminary  Sentence:  The  guerrillas  attacked  the  farms.) 

The  Santo  Tomas  farm  is  the  farm  with  damage. 

Pattern:  [[NP]  with  [NP]]  [VP]. 

Example:  The  Governor  with  Marxist  ideals  was  kidnapped  by  terrorists. 

Criteria:  Demonstrated  understanding  of  the  possessive  relationship  between  the  noun 
phrase  preceding  the  “with”  and  the  noun  phrase  following  it. 

_ Score: _ 


5.4  Combinations  of  Genitives  and  Alternatives 
5.4.1  Genitive  and  OE-Phrase 

In  this  combination,  the  noun  phrase  to  be  marked  with  the  genitive  case  marking  will  consist 
of  a  noun  phrase  postmodified  by  a  prepositional  phrase  o/-phrase. 

The  genitive  marking  in  the  construction  “[[NP]  o/[NP]  -(-genitive]  [NP]”  marks  the  “[NP] 
o/[NP]”.  For  example,  in  the  sentence  fragment: 

the  oldest  employee  of  the  company’s  salary 

the  genitive  marks:  “the  oldest  employee  of  the  company 


Test  the  o/-phrase  as  part  of  a  noun  phrase  which  is  case-marked  by  the  genitive. 

Pattern:  List  [[NP]  of  [NP]  -(-genitive]  [NP]. 

Example:  List  the  oldest  employee  of  the  company’s  salary. 

Pattern:  [WH-Word]  ([NP])  [BE- Verb]  [[NP]  of  [NP]  +genit.ve]  [NP]  ? 

Example:  What  is  the  president  of  the  company’s  salary  ? 

Pattern:  [NP]  [Verb]  [[NP]  of  [NP]  +genjt.ve]  [NP]. 

Example:  Terrorists  kidnapped  the  son  of  Cuilapa’s  mayor. 

Pattern:  [[NP]  of  [NP]  -(-genitive]  [NP]  [VP], 

Examples:  The  son  of  Cuilapa’s  mayor  was  kidnapped  by  terrorists. 

The  manager  of  President  Cerezo’s  farm  was  killed  in  the 

guerrilla  attack. 

Criteria:  Demonstrated  understanding  of 

•  the  relationship  between  the  noun  phrase  following  the  “of”  and  the  noun  phrase 
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preceding  it. 

•  the  relationship  between  [[NP]o/[NP]  -f  genitive]  and  the  noun  phrase  immediately 
following. 

_ Score: _ 


5.4.2  Combination  of  the  OF-Phrase  and  the  W/77/-Phrase 

In  this  combination,  a  noun  phrase  will  be  postmodified  by  the  prepositional  o/-phrase. 

The  noun  phrase  in  the  o/-phrase  will  itself  be  postmodified  by  the  prepositional  phrase 
tmf/i-phrase. 

Test  the  o/-phrase  as  part  of  a  noun  phrase  which  is  case-marked  by  the  genitive. 

Pattern:  List  [NP]  of  [[NP]  with  [NP]]. 

Example:  List  the  name  of  the  employee  with  the  highest  salary. 

Pattern:  [WH-Word]  ([NP])  [BE- Verb]  [NP]  of  [[NP]  with  [NP]]  ? 

Example:  What  is  the  sales  of  the  employee  with  the  highest  sal 

Pattern:  [NP]  [Verb]  [NP]  of  [[NP]  with  [NP]]. 

Example:  Terrorists  kidnapped  the  son  of  the  Governor  with  Marxist  ideals. 

Pattern:  [NP]  of  [[NP]  with  [NP]]  [VP]. 

Example:  The  son  of  the  Governor  with  Marxist  ideals  was  kidnapped  by  terrorists. 

Criteria:  Demonstrated  understanding  that 

•  the  noun  phrase  following  the  preposition  “of”  in  the  o/-phrase  is  postmodified  by 

the  with-phrase 

•  the  complete  o/-phrase,  including  its  postmodification,  postmodifies  the  noun  phrase 

which  immediately  preceded  it. 

_ Score: _ 


6.  Postmodification 
6.1  Relative  Clauses 

A  relative  clause  is  an  embedded  sentence  that  is  found  in  the  postmodification  position  of 
a  noun  phrase.  A  full  relative  clause  consists  of  a  relative  pronoun  followed  by  a  sentence  or 
verb  phrase  with  some  omitted  constituent(s).  For  example,  the  sentence 

The  plane  [that  we  saw]  was  a  DC- 10 

includes  the  relative  clause  “that  we  saw”  (in  brackets).  This  relative  clause  consists  of  the 
relative  pronoun  “that”  followed  by  the  sentence 
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we  saw  (the  plane) 
where  “the  plane”  has  been  omitted. 

Relative  pronouns  have  the  double  role  of  referring  to  its  antecedent ,  the  head  of  the  noun 
phrase  being  modified,  and  of  filling  a  grammatical  function  in  the  relative  clause  (that  of 
the  omitted  phrase).  For  example,  in: 

The  woman  [ who  hired  Mark  Jones]  is  a  PhD. 

“who”  refers  to  the  woman,  and  functions  grammatically  as  the  subject  in  the  relative  clause. 
In: 


The  man  [whom  we  saw]  was  the  President. 

“whom”  refers  to  the  man,  and  functions  grammatically  as  the  object  in  the  relative  clause. 

Relative  pronouns  are  categorized  as  personal  or  non-personal.  The  personal  pronouns  are 
“that”,  “who”,  “whose”,  “whom”  and  the  non-personal  pronouns  are  “which”  and,  again, 
“that”  and  “whose”.  Personal  relative  pronouns  are  used  to  refer  to  human  beings  as  well 
as  other  entities  that  are  thought  to  have  human  characteristics  or  are  thought  of  in  a 
human-like  role.  For  example,  in  the  sentences: 

The  angel  [who  was  making  faces]  had  a  tilted  halo 
Rover,  [who  was  barking],  frightened  the  children. 

the  personal  pronoun  is  used  to  refer  to  an  angel,  or  a  pet.  Non-personal  pronouns  are  used 
to  refer  to  non-human  objects: 

The  plane  [which  we  s<m]  was  a  DC- 10 

and  they  may  refer  to  collective  entities  that  are  composed  of  humans: 

The  committee  [which  is  responsible  for  facilities]  has  postponed  a  decision. 

When  the  relative  pronoun  is  used  as  an  object  in  the  embedded  sentence,  it  may  be  omitted 
from  the  relative  clause.  Thus  the  relative  pronoun  can  be  omitted  from  the  relative  clause 
“which  we  saw”,  in  the  sentence: 

The  plane  [which  we  saw]  was  a  DC-10 

yielding  the  equivalent  sentence: 

The  plane  [we  saw]  was  a  DC-10. 
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Relative  clauses  are  of  two  types:  restrictive  and  non-restrictive.  Restrictive  relative  clauses 
impose  a  limitation  on  the  referent  of  the  antecedent,  as  in: 

The  person  [that  is  head  of  the  C.S.  Department]  has  a  Ph.D.  from  Princeton. 

Non-restrictive  relative  clauses  are  parenthetic  comments  which  provide  additional  informa¬ 
tion  about  the  referent  of  the  antecedent,  but  do  not  further  restrict  the  referent,  as  in: 

The  head  of  the  Accounting  Department,  [u>/io  previously  worked  for  Unisys],  has 
been  promoted  to  V.P.  for  Finance. 

The  relative  pronouns  “who”,  “whose”,  “whom” ,  and  “which ”  can  be  used  in  either  restrictive 
or  non-restrictive  relative  clauses,  but  the  relative  pronoun  “that”  is  only  used  in  restrictive 
relative  clauses  and  is  usually  the  preferred  choice. 

The  following  subsections  consist  of  test  items  that  cover  the  issues  discussed  above.  Be 
sure  to  use  sentence  types  that  the  system  has  successfully  processed.  The  only  new  element 
to  be  added  to  a  previously  tested  sentence  is  the  relative  clause  as  postmodifier  of  a  noun 
phrase.  Refer  back  to  the  beginning  of  this  current  Section  III,  titled  “Noun  Phrases”,  for 
the  structure  of  the  basic  sentence  patterns. 

6.1.1  Relative  Pronoun  as  Subject 

The  structure  of  this  type  of  relative  clause  is: 

Pattern:  [Rel  Pronoun]  [VP] 

Example:  that  hired  Mary  Smith 

who  joined  the  C.S.  Department 

with  the  relative  pronoun  functioning  as  the  grammatical  subject.  Note  that  the  relative 
pronoun  acts  as  a  [NP]  preceding  the  [VP]. 

In  the  next  three  test  items,  use  this  type  of  relative  clause  in  the  postmodification  position 
of  a  noun  phrase. 

6. 1.1.1  The  Relative  Pronoun  THAT  -  Restrictive  Only 

Test  “that”  as  the  relative  subject  pronoun  in  a  relative  clause  postmodifying  a  noun  phrase. 

Eg,  Is  John  Smith  the  person  [that  is  V.P.  of  Finance]  ? 

Eg,  Who  is  the  person  [that  is  V.P.  of  Finance]  ? 

Eg,  List  the  person  [that  is  V.P.  of  Finance]. 

Eg,  The  terrorists  [that  attacked  the  Government  house]  killed 
three  civilians. 

Eg,  Terrorists  killed  the  official  [that  was  kidnapped] . 

Eg,  The  guerrillas  [that  attacked  the  presidential  farm]  killed  a 
security  guard. 
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Criteria:  Demonstrated  understanding  of  the  qualification  imposed  on  the  noun  phrase  by 
the  immediately  following  relative  clause. 

_ Score:  _ 


6. 1.1. 2  Personal  Relative  Pronoun  WHO 

Test  “who”  as  the  relative  subject  pronoun  in  a  relative  clause  postmodifying  a  noun  phrase. 

Eg,  Was  the  man  [vho  boarded  the  plane  in  Syracuse]  a  professor  ? 
Eg,  Is  the  person  [vho  is  head  of  Department  77]  a  male  ? 

Eg,  Is  John  Smith  the  person  [vho  is  head  of  Department  77]  ? 

Eg,  The  terrorists  [vho  attacked  the  Government  house]  killed 
three  civilians. 

Eg,  Terrorists  killed  the  official  [vho  vas  kidnapped] . 

Eg,  The  guerrillas  [vho  attacked  the  presidential  farm]  killed  a 
security  guard. 

Criteria:  Demonstrated  understanding  of  the  qualification  imposed  on  the  noun  phrase  by 
the  immediately  following  relative  clause  with  personal  relative  pronoun. 

- - - - - Score: _ 


6. 1.1. 3  Non-Personal  Relative  Pronoun  WHICH 

Test  “which”  as  the  relative  subject  pronoun  in  a  relative  clause  postmodifying  a  noun  phrase. 

Eg,  Was  the  plane  [vhich  flev  to  Syracuse]  a  DC- 10  ? 

Eg,  Is  D77  the  department  [vhich  includes  John  Smith]  ? 

Eg,  The  terrorist  group  [vhich  attacked  the  Government  house] 
killed  three  civilians. 

Eg,  ARENA  vas  responsible  for  the  bombing  of  the  government 
house  [vhich  killed  three  people] . 

Eg,  The  guerrilla  group  [vhich  attacked  the  presidential  farm] 
killed  a  security  guard. 

Criteria:  Demonstrated  understanding  of  the  qualification  imposed  on  the  noun  phrase  by 
the  immediately  following  relative  clause  with  non-personal  relative  pronoun. 

- — - — - Score: _ 


6.1.2  Possessive  Relative  Pronoun  (WHOSE)  as  a  Determiner 

The  structure  of  this  type  of  relative  clause  is: 

Pattern:  [Rel  Pronoun]  [NP  — DET  1  [VP] 

Example:  whose  salary  exceeds  $35,000 
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with  the  relative  pronoun  it  WHOSE  functioning  as  the  determinative  element  of  the  noun 
phrase  which  is  the  grammatical  subject  of  the  relative  clause. 

Eg.  List  the  employees  [whose  field  is  Computer  Science] . 

Eg,  List  the  employees  [whose  salary  exceeds  $35,000] . 

Eg,  Terrorists  attacked  peasants  [whose  land  was  in  their  territory]. 
Eg,  The  guerrillas  attacked  the  president  [whose  farm  is  the  Santo 
Tomas  farm] . 


Criteria:  Demonstrated  understanding: 

e  that  the  whose  functions  as  a  determiner  in  the  relative  clause  which  it  begins, 
e  of  the  qualification  imposed  on  the  noun  phrase  postmodified  by  the  relative  clause 
introduced  by  whose. 

_ Score: _ 


6.1.3  Relative  Pronoun  as  Object 

The  structure  of  this  type  of  relative  clause  is: 

Pattern:  [Rel  Pronoun]  [NP]  [Verb] 

Examples:  that  James  Harris  hired 

who  the  Chairman  promoted 

with  the  relative  pronoun  functioning  as  the  (otherwise  omitted)  grammatical  object.  Note 
that  there  is  no  [NP]  following  the  [Verb]. 

In  the  next  three  test  items,  use  this  type  of  relative  clause  in  the  postmodification  position 
of  a  noun  phrase. 

6. 1.3.1  The  Relative  Pronoun  THAT  as  Object  -  Restrictive  Only 

Test  “that”  as  the  relative  object  pronoun  in  a  relative  clause  postmodifying  a  noun  phrase. 

Eg,  Is  John  Smith  the  person  [that  the  V.P  of  Finance  fired]  ? 

Eg,  The  Government  house  [that  ARENA  attacked]  was  completely 
destroyed . 

Eg,  The  farm  [that  the  guerrillas  attacked]  was  the  presidential 
farm. 


Criteria:  Demonstrated  understanding  that  the  noun  phrase  being  postmodified  expresses 
the  object  of  the  event  or  relationship  specified  by  the  immediately  following  relative  clause. 


Score: 
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6. 1.3.2  Personal  Relative  Pronoun  WHOM  as  Object 

Test  “whom”  as  the  relative  object  pronoun  in  a  relative  clause  postmodifying  a  noun  phrase. 

Eg,  Was  the  man  [whom  we  saw]  a  professor  ? 

Eg,  The  government  official  [whom  terrorists  killed]  was  the 
Honduran  Prime  minister. 

Eg,  The  govemaent  official  [whom  the  guerrillas  attacked]  was 
President  Cerezo. 

Criteria:  Demonstrated  understanding  that  the  noun  phrase  being  postmodified  expresses  a 
person  who  is  the  object  of  the  event  or  relationship  specified  by  the  immediately  following 
relative  clause. 

_ Score: _ 


6. 1.3.3  Non-Personal  Relative  Pronoun  WHICH  as  Object 

Test  “which”  as  the  relative  object  pronoun  in  a  relative  clause  postmodifying  a  noun  phrase. 

Eg,  Was  the  plane  [vhich  ve  sav]  a  DC- 10  ? 

Eg,  Fifty  government  employees  died  in  the  Government  house 
[which  terrorists  attacked] . 

Eg,  The  farm  vhich  the  guerrillas  attacked  was  the  presidential 
farm. 


Criteria:  Demonstrated  understanding  that  the  noun  phrase  being  postmodified  expresses 
a  (non-human)  object  of  the  event  or  relationship  specified  by  the  immediately  following 
relative  clause. 


Score: 


6.1.4  Deletion  of  Relative  Pronoun  Object. 

This  type  of  relative  clause  consists  of  a  noun  phrase  followed  by  a  verb  phrase  with  its  rela¬ 
tive  pronoun  deleted  or  omitted.  The  relative  pronoun  which  was  used  in  the  initial  position 
in  the  relative  clauses  discussed  in  the  preceding  sections  has  been  deleted  or  omitted. 

Pattern:  [NP]  [Verb] 

Examples:  John  Smith  hired 

the  department  head  promoted 

Use  a  test  input  with  this  type  of  relative  clause  in  the  postmodifi cation  position  of  a  noun 
phrase.  Be  sure  that  the  noun  phrase  being  postmodified  expresses  the  object  of  the  relative 
clause. 


Eg,  Was  the  plane  [ve  sav]  a  DC- 10  ? 
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Eg,  Is  D77  the  department  [John  Smith  administers]  ? 

Eg,  The  Honduran  Prime  minister  vas  the  government  official 
[terrorists  killed] . 

Eg,  The  Government  house  [terrorists  bombed]  vas  completely 
destroyed. 

Eg,  The  farm  [the  guerrillas  attacked]  vas  the  presidential 
farm. 


Criteria:  Demonstrated  understanding  that  the  noun  phrase  being  postmodified  expresses 
the  object  of  the  event  or  relationship  specified  by  the  immediately  following  relative  clause. 

_ Score: _ 


6.1.5  Relative  Pronoun  Prepositional  Object 

Often,  a  relative  pronoun  functions  as  the  object  of  a  preposition,  as  in  the  sentences: 

The  man  [for  whom  Jim  Davis  works]  is  a  lawyer. 

The  man  [who  Jim  Davis  works  for]  is  a  lawyer. 


6. 1.5.1  Preposition  First  in  the  Relative  Clause 

The  structure  of  this  type  of  relative  clause  is: 

Pattern:  [Preposition]  [Rel  Pronoun]  [NP]  [Verb] 

Example:  for  whom  Jim  Davis  works 

Use  a  test  input  with  this  type  of  relative  clause  in  the  postmodification  position  of  a  noun 
phrase. 


Eg,  Is  John  Smith  a  person  [to  vhom  a  salary  is  paid]  ? 

Eg,  List  the  employees  [for  vhom  John  Smith  is  supervisor] . 

Eg,  The  terrorist  leader  [for  vhom  ARENA  kidnapped  the  Prime 
Minister]  assassinated  the  Prime  Minister. 

Eg,  The  government  official  [against  vhom  the  URNG  guerrillas 
conducted  an  attack]  vas  President  Cerezo. 

Eg,  Has  the  plane  [of  vhich  ve  spoke]  a  VSTOL  aircraft  ? 

Eg,  Is  D77  the  department  [of  vhich  John  Smith  is  a  member]  ? 

Eg,  Is  D77  the  department  [of  vhich  Mary  Jones  is  head]  ? 

Eg,  The  rightving  terrorist  alliance  [for  vhich  ARENA  kidnapped 
the  Honduran  Prime  Minister]  attacked  Government  houses  in 
four  countries. 

Eg,  The  farm  [against  vhich  the  URNG  guerrillas  conducted  an 
attack]  vas  the  Santo  Tomas  farm. 
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Criteria:  Demonstrated  understanding  of  the  qualification  imposed  on  the  referent  of  the 
noun  phrase  by  the  postmodifying  relative  clause  introduced  by  the  preposition. 

_ Score: _ 


6. 1.5. 2  Preposition  Following  the  Verb  in  the  Relative  Clause 

A  frequently  occurring  form  of  the  relative  pronoun  acting  as  the  object  of  a  preposition, 
has  the  preposition  following  the  verb.  The  structure  of  this  type  of  relative  clause  is: 

Pattern:  [Rel  Pronoun]  [NP]  [Verb]  [Preposition] 

Example:  whom  Jim  Davis  works  for 

Use  a  test  input  with  this  type  of  relative  clause  in  the  postmodification  position  of  a  noun 
phrase. 


Eg,  Was  the  man  [whom  ve  spoke  of]  a  professor  ? 

Eg,  Was  the  man  [whom  ve  spoke  of]  a  professor  ? 

Eg,  Is  John  Smith  the  person  [whom  Tom  Harris  works  for]  ? 

Eg,  The  terrorist  leader  [whom  ARENA  kidnapped  the  Prime 
Minister  for]  assassinated  the  Prime  Minister. 

Eg,  The  government  official  [whom  the  URNG  guerrillas 
conducted  an  attack  against]  was  President  'Cerezo. 

Eg,  Was  the  plane  [which  we  spoke  of]  a  VSTOL  aircraft  ? 

Eg,  Is  D77  the  department  [that  John  Smith  is  a  member  of]  ? 

Eg.  The  rightwing  terrorist  alliance  [which  ARENA  kidnapped  the 
Honduran  Prime  Minister  for]  attacked  Government  houses  in 
four  countries. 

Eg,  The  farm  [which  the  URNG  guerrillas  conducted  an  attack 
against]  was  the  Santo  Tomas  farm. 


Criteria:  Demonstrated  understanding  of  the  qualification  imposed  on  the  referent  of  the 
noun  phrase  by  the  postmodifying  relative  clause. 

- Score: _ 


6.1.6  Relative  Pronoun  as  Adverbial 

A  relative  pronoun  as  the  object  of  a  preposition  can  function  as  an  adverbial.  For  adverbial 
expressions  of  time,  place,  and  cause,  the  preposition  and  following  relative  pronoun  can  be 
replaced  by  special  adverbs  such  as  u where ”,  “when”,  “why”. 

The  structure  of  a  relative  clause  with  relative  pronoun  as  adverbial  is: 
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Pattern: 

(Preposition]  [Rel  Pronoun] 

[NP] 

[VP] 

UT 

[WH-Adverb] 

[NP] 

[VP] 

Place: 

Examples: 

at  which 

John  Smith 

works 

where 

John  Smith 

works 

Time: 

Examples: 

during  which 

John  Smith 

works 

where 

John  Smith 

works 

6. 1.6.1  Relative  Pronoun  as  Adverbial  for  Place 

6. 1.6. 1.1  With  Preposition  and  Relative  Pronoun 

Eg,  List  the  address  [at  vhich  John  Smith  resides] . 

Eg,  The  country  [in  which  terrorists  bombed  the  Government  house] 
is  Honduras. 

Eg,  The  city  [in  which  the  guerrillas  robbed  peasants]  is  Flores. 


Criteria:  Demonstrated  understanding  of  the  locative  qualification  imposed  on  the  noun 
phrase  by  the  immediately  following  relative  clause. 

_ Score: _ 


6. 1.6. 1.2  With  WHERE 

Eg,  List  the  address  [where  John  Smith  resides] . 

Eg,  The  country  [where  terrorists  bombed  the  Government  house] 
is  Honduras. 

Eg,  The  city  [where  the  guerrillas  robbed  peasants]  is  Flores. 


Criteria:  Demonstrated  understanding  of  the  locative  qualification  imposed  on  the  noun 
phrase  by  the  immediately  following  relative  clause. 

_ _ Score: _ 


6. 1.6. 2  Relative  Pronoun  as  Adverbial  for  Time 
6.1. 6.2.1  With  Preposition  and  Relative  Pronoun 

Eg,  List  the  date  [on  which  John  Smith  was  hired] . 

Eg,  The  date  [on  which  terrorists  bombed  the  Government  house] 
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was  December  15. 

Eg,  The  date  [on  which  the  guerrillas  attacked  the  farm]  is 
February  2. 

Criteria:  Demonstrated  understanding  of  the  temporal  qualification  imposed  on  the  noun 
phrase  by  the  immediately  following  relative  clause. 

_ Score: _ 


6.1. 6.2.1  With  WHEN 

Eg,  List  the  date  [when  John  Smith  was  hired] . 

Eg,  The  date  [when  terrorists  bombed  the  Government  house] 
was  December  15. 

Eg,  The  date  [when  the  guerrillas  attacked  the  farm]  is 
February  2. 

Criteria:  Demonstrated  understanding  of  the  temporal  qualification  imposed  on  the  noun 
phrase  by  the  immediately  following  relative  clause. 

_ Score: _ 


6. 1.6. 3  Omitted  Relative  Pronoun  -  Adverbial  Case 

Eg,  List  the  date  [John  Smith  was  hired] . 

Eg,  List  the  place  [John  Smith  works] . 

Eg,  The  date  [terrorists  bombed  the  Government  house]  was 
December  15. 

Criteria:  Demonstrated  understanding  of  the  temporal  or  locative  qualification  imposed  on 
the  noun  phrase  by  the  immediately  following  relative  clause. 

_ Score: _ 


6.1.7  Multiple  Relative  Clauses  Within  the  Same  Noun  Phrase 

A  noun  phrase  can  have  more  than  one  relative  clause  in  postmodifying  one  noun  phrase. 
The  structure  of  this  type  of  noun  phrase  is 

Pattern:  ([Det])  ([Premodification])  [Head]  [Relative  Clause]  [Relative  Clause] 

Example:  the  employee  D77  hired  that  worked  for  IBM. 
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Eg,  Is  John  Smith  the  employee  [D77  hired]  [that  worked  for  IBM] . 
Eg,  The  terrorist  [police  captured]  [who  attacked  the  Government 
house]  killed  three  civilians. 

Criteria:  Demonstrated  understanding  of  the  qualifications  imposed  on  the  noun  phrase  by 
the  relative  clauses  which  postmodified  it. 

_ Score: _ 
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6.2  Reduced  Relative  Clauses 

Reduced  relative  clauses  are  postmodifiers  formed  by  a  relative  clause  from  which  the  relative 
pronoun  and  the  auxiliary  BE- Verb  have  been  deleted. 


Pattern: 

[NP] 

[Postmodification] 

Examples: 

Subject  Compl.  Clause 

Full: 

My  brother, 

who  is  an  engineer, 

as  Postmodifier 

Reduced: 

My  brother, 

an  engineer, 

Progressive  Clause 

Full: 

the  man 

who  is  reading  a  book 

as  Postmodifier 

Reduced: 

the  man 

reading  a  book 

Passive  Clause 

Full: 

the  employee 

who  was  hired  yesterday 

as  Postmodifier 

Reduced: 

the  employee 

hired  yesterday 

6.2.1  Subject  Complement  Clauses 

In  this  subsection,  test  reduced  relative  clauses  using  a  noun  phrase  as  subject  comple¬ 

ment.  A  full  relative  clause  consisting  of  a  copular  sentence  such  as  “who  is  an  engineer ”  is 

transformed  into  the  noun  phrase  “an 

i  engineer”  as 

a  reduced  relative. 

Pattern: 

[NP] 

[Postmodification] 

Examples: 

Full: 

My  brother, 

who  is  an  engineer, 

Reduced: 

My  brother, 

an  engineer, 

Use  a  noun  phrase  as  a  reduced  relative  clause  to  provide  additional  description  or  specifica¬ 
tion  of  the  noun  phrase  that  it  immediately  follows  as  a  postmodifier.  For  more  information 
on  subject  complements,  see  Simple  Verb  Phrases  in  Section  I,  Basic  Sentences. 


A  noun  phrase  with  this  type  of  postmodification  looks  like: 


Pattern: 

Examples: 


|NP] 

[([Det])  ([Premodification])  [Head]] 

John  Smith, 

the  new  president, 

ARENA, 


[Subject  Comp.  Clause] 

[Subject  Complement] 
an  employee 
Mary  Jones 
a  terrorist  organization 


or  more  simply,  a  second  noun  phrase  directly  following  the  head  of  the  postmodified  noun 
phrase. 


Full  sentence  examples  with  the  reduced  relative  clauses  bracketed  for  ease  of  identification: 


Eg,  List  the  salary  of  John  Smith,  [an  employee]. 

Eg,  ARENA,  a  terrorist  organization,  [killed  three  civilians] . 
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Criteria:  Demonstrated  understanding  of  the  qualification  imposed  on  the  referent  of  the 
noun  phrase  by  the  reduced  relative  clause  which  immediately  follows  it  as  a  postmodifier. 

_ _ Score: _ 


6.2.1  Passive  Clauses 

A  passive  clause  consists  of  a  verb  phrase  with  a  passive  participle  (for  regular  verbs,  a 
passive  participle  is  the  form  of  a  verb  with  an  “-ed”  or  “-en”  ending)  as  the  head  of  the 
verb  phrase.  The  verb  may  be  a  transitive  verb  followed  by  a  noun  phrase,  or  it  may  be  an 
intransitive  verb.  Test  a  passive  clause  as  postmodifier  of  a  noun  phrase.  A  noun  phrase 
with  this  type  of  postmodification  looks  like: 

Pattern:  [NP]  [Passive  Clause] 

[([Det])  ([Premodification])  [Head]  ]  [[Verb  4-passive]  ([NP])] 

Example:  the  employee  paid  $35,000 

Full  sentence  examples  with  the  reduced  relative  clauses  bracketed  for  ease  of  identification: 

Eg.  List  the  employees  [paid  $35,000]. 

Eg.  List  the  female  employees  [paid  $40,000]. 

Eg,  The  Mayor  [kidnapped  by  ARENA]  died. 

Criteria:  Demonstrated  understanding  of  the  qualification  imposed  on  the  referent  of  the 
noun  phrase  by  the  passive  clause  which  immediately  follows  it  as  a  postmodifier. 

_ Score: _ 


6.2.2  Progressive  Clauses 

A  progressive  clause  consists  of  a  verb  phrase  with  a  progressive  participle  (for  regular  verbs, 
a  progressive  participle  is  the  form  of  a  verb  with  an  “-mg”  ending)  as  the  head  of  the  verb 

phrase.  The  verb  may  be  a  transitive  verb  followed  by  a  noun  phrase,  or  it  may  be  an 

intransitive  verb.  Test  a  progressive  clause  as  postmodifier  of  a  noun  phrase.  A  noun  phrase 
with  this  type  of  postmodification  looks  like: 

Pattern:  [NP]  [Progressive  Clause] 

[  ([Det])  ([Premodification])  [Head]]  [[Verb  +prog]  ([NP])  ] 

Example:  the  new  employee  selling  CPUs. 

Full  sentence  examples  with  the  reduced  relative  clause  bracketed  for  ease  of  identification: 

Eg,  List  the  employees  [selling  CPUs] . 

Eg,  The  terrorists  [attacking  the  government  house]  killed  the  Mayor. 

Criteria:  Demonstrated  understanding  of  the  qualification  imposed  on  the  referent  of  the 
noun  phrase  by  the  progressive  clause  which  immediately  follows  it  as  a  postmodifier. 

_ Score: _ 
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6.2.3  Infinitive  Clauses 


An  infinitive  clause  consists  of  the  infinitive  verb  (an  unmarked  verb  following  the  word  “to”) 
as  the  head  of  a  verb  phrase.  It  may  be  a  transitive  verb  followed  by  a  noun  phrase,  or  it 
may  be  an  intransitive  verb.  Test  an  infinitive  clause  as  a  postmodifier  of  a  noun  phrase.  A 
noun  phrase  with  this  type  of  postmodification  looks  like: 

Pattern:  [NP]  [Infinitive  Clause] 

[  ([Det])  ([Premodification])  [Head]]  [  [Verb  +inf]  ([NP])] 

Example:  the  employee  to  sell  cpus. 

Eg,  List  the  names  of  the  employees  [to  earn  $35,000]. 

Eg,  The  terrorists  [to  be  arrested]  killed  the  Mayor. 

Criteria:  Demonstrated  understanding  of  the  qualification  imposed  on  the  referent  of  the 
noun  phrase  by  the  infinitive  clause  which  immediately  follows  it  as  a  postmodifier. 

_ _ Score: _ 
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IV.  ADVERBIALS 


Following  usual  principles  of  classification,  such  as  the  similarity  of  word  meaning  within  a 
class  and  where  these  words  may  appear,  adverbials  are  the  most  heterogeneous  word  class. 

Words  or  phrases  used  to  modify  anything  other  than  a  noun  have  traditionally  been  called 
adverbs  or  adverbials.  Adverbs  and  adverbials  can  modify  adjectives,  other  adverbs,  verb 
phrases,  clauses,  and  even  sentences.  The  term  “adverbials”  refers  to  both  single  words 
adverbs  as  well  as  multi-word  phrases. 

Adverbials  (including  adverbs)  are  traditionally  classified  according  to  form  (how  they  are 
composed),  function  (what  part  of  a  sentence  or  text  they  modify),  semantics  (the  meaning 
expressed  by  the  adverbial),  and  position  (where  in  a  sentence  the  adverbial  occurs). 

There  are  correspondences  between  subclasses  of  adverbials  classified  by  one  criteria  and 
subclasses  classified  by  other  criteria.  For  example,  the  semantic  category  attitudinal 
adverbials  includes  those  adverbials  which  express  the  speaker’s  attitude.  The  adverbials 
in  this  group  will  functionally  be  defined  as  meta-sentential  adverbials  and  will  give 
information  outside  the  scope  of  the  events  described  by  the  sentence. 

In  this  section  we  will  only  test  adverbials  according  to  form  and  position ,  but  not  according 
to  function  or  semantics.  With  regard  to  semantics,  adverbials  express  a  range  of  elements 
which  we  conceptualize  as  part  of  an  event,  for  instance  time  or  space  or  manner.  In  Section 
XII  Semantics  of  Events ,  we  discuss  and  test  some  of  the  standard  semantic  elements  of  an 
event  including  those  expressed  by  adverbials. 


Classification  According  to  Form 

Adverbials  may  be  classified  by  form,  that  is,  how  they  are  composed,  as  follows: 

Simple  Adverbs  are  single  word  items  which  include: 

derivational  adverbs,  that  is,  adverbs  formed  by  the  process  of  suffixing  -ly  to  an 
adjective  stem,  as  in  quickly  from  quick ,  or  carefully  from  careful. 

non-derivational  adverbs,  including  words  which  are  also  adjectives,  such  as  fast 
or  loud ,  or  lexical  items  such  as  well,  very,  how,  just,  so,  soon,  yet  and  so  on. 

pronominal  adverbs  including: 

demonstrative:  here,  there,  now,  then,  . . . 
interrogative:  where,  when,  how,  why,  . . . 

indefinite:  somewhere,  anywhere,  everywhere,  nowhere,  ever,  never,  always, 

Prepositional  phrases  may  be  used  as  adverbials.  Examples  are  : 

I  work  in  a  factory. 

Until  8  o  ’ clock  I’ll  be  at  the  store. 
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Noun  phrases  are  sometimes  used  as  adverbials.  Examples  are  : 

I  worked  yesterday. 

I  am  going  to  London  this  year. 

Finite  clauses  can  also  be  used  as  adverbials.  Finite  clauses  are  subordinate  clauses  having 
a  finite  verb,  i.e.,  one  that  is  tensed  and  inflected  for  person.  Examples  are: 

David  plays  chess  as  his  father  taught  him. 

David  gave  William  money  whenever  he  needed  it. 

Non-finite  clauses,  clauses  with  a  non-finite  verb  phrase,  cam  function  as  adverbials.  They 
are  classified  according  to  the  three  non-finite  verb  types: 

to-infinitive,  e.g.: 

David  plays  chess  to  please  his  father. 

-ing  participle,  e.g.: 

Standing  on  a  hill,  they  fought  their  last  battle. 

-ed  participle,  e.g.: 

Whenever  approached  by  him ,  David  gave  William  money. 

Verbless  clauses  do  not  include  a  verb,  but  can  nonetheless  be  analyzed  as  a  clause 
with  a  missing  be  and,  often,  a  missing  subject.  Examples  of  verbless  clauses  used  as 
adverbials  are: 

David  plays  chess  when  on  holiday.  [when  he  is  on  holiday] 

Fearless,  they  fought  their  last  battle.  [Being  fearless] 

Though  indignant  at  his  threats,  David  gave  William  money.  [Though  he  was  indignant  . . 

Classification  According  to  Position 

Given  the  fairly  fixed  word  order  in  English,  adverbials  are  the  most  mobile  of  all  sentence 
elements  and  may  occur  in  any  of  the  following  positions: 

At  the  beginning  of  the  sentence  (pre-subject  or  initial  position ): 

Frankly,  I  don’t  care  for  caviar. 

Standing  on  a  hill,  they  fought  their  last  battle. 

Between  the  subject  and  verb  (pre-verb  position): 

John  carefully  opened  the  door. 

He  probably  didn’t  realize  what  he  was  doing. 

Between  the  auxiliary  and  the  main  verb: 
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I  should  never  have  thought  of  that. 

I  should  have  never  thought  of  that. 

Between  the  verb  and  its  direct  or  indirect  object: 

He  spoke  kindly  to  me. 

She  wrote  in  feverish  rage  long  violent  letters  of  complaint.  [Quirk  et 
al,  1985] 

At  the  end  of  the  sentence  ( final  position ): 

The  trip  was  quite  enjoyable  after  all. 

I  work  in  a  factory. 


Not  every  adverbial  can  occur  in  every  position.  Furthermore,  an  adverbial  in  one  position 
may  have  a  different  meaning  or  function  than  the  same  word  or  phrase  in  a  different  position. 
For  example, 

Clumsily,  John  stepped  on  the  snail. 

is  interpreted  as  expressing  the  speaker’s  attitude  that  it  was  clumsy  of  John  to  step  on  the 
snail,  while 

John  stepped  on  the  snail  clumsily. 

is  interpreted  as  expressing  the  way  in  which  John  snail-stepped.  The  sentence 
John  clumsily  stepped  on  the  snail, 
is  ambiguous  between  the  above  two  meanings. 


1  Adverbs  and  Adverbials 

In  this  subsection,  we  are  testing  simple  adverbs  and  prepositional  phrases  as  adverbials  in 
initial,  pre-verb,  and  final  positions. 

1.1  Adverbials  in  Initial  Position 

In  the  following  subsections,  we  will  test  adverbials  in  the  initial  position.  Below  are  examples 
of  adverbials  in  initial  position.  The  adverbials  are  bracketed  for  ease  of  identfication. 

Eg,  [In  1989]  who  was  hired? 

Eg,  [Last  year]  vhat  employee  received  a  raise? 

Eg,  [Yesterday]  terrorists  bombed  the  Bank  of  Colombia. 

Eg,  [On  December  15]  assassins  kidnapped  the  Prime  Minister. 
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1.1.1  Simple  Adverbs  in  Initial  Position 

In  the  following  example,  the  word  “yesterday”  is  a  simple  adverb  in  initial  position.  I  i 
demonstrating  understanding  of  the  sentence: 

Yesterday  who  had  a  birthday? 

an  employee  database  would  be  expected  to  calculate  yesterday’s  date,  relative  to  the  sys¬ 
tem’s  current  date,  and  compare  the  day  to  its  databank  of  employees’  birthdays. 

A  MUC  system,  given  the  sentence: 

Today  terrorists  bombed  the  Bank  of  Columbia. 

would  be  expected  to  create  a  template  which  reflects  that  a  terrorist  bombing  of  the  Bank 
of  Columbia  occurred  on  the  date  that  the  system  considers  to  be  the  current  date. 

Use  any  sentence  pattern(s)  from  previous  sections  which  the  system  has  successfully  pro¬ 
cessed.  Add  an  [Adverbial]  to  the  beginning  of  the  sentence  pattern(s).  Some  examples  are 
shown  below.  Now  replace  the  [Adverbial]  in  the  sentence  pattern  with  a  simple  adverb. 


Pattern: 

[Adverbial] 

[WH-Word] 

([NP])  [VP]? 

Example: 

Yesterday 

who 

had  a  birthday? 

Pattern: 

[Adverbial] 

[NP] 

[VP]. 

Examples: 

Yesterday 

terrorists 

bombed  the  Bank  of  Colombia. 

Today 

assassins 

kidnapped  the  Prime  Minister. 

Today 

the  guerrillas 

;  attacked  the  presidential  farm. 

Criteria:  Demonstrated  understanding  of  the  meaning  expressed  by  the  simple  adverb  in 
initial  position. 


Score: 


1.1.2  Prepositional  Phrases  as  Adverbials  in  Initial  Position 


Use  any  sentence  pattern(s)  from  previous  sections  which  the  system  has  successfully  pro¬ 
cessed.  Add  an  [Adverbial]  to  the  beginning  of  the  sentence  pattern(s).  Some  examples 
are  shown  below.  Now  replace  the  [Adverbial]  in  the  sentence  pattern  with  a  prepositional 
phrase  ([PP]). 


Pattern: 

Examples: 


[Adverbial]  [WH-Word] 

In  1989  who 

In  the  New  York  office  what 


«NP]) 

employee 


[VP]? 


sold  CPUs? 
received  a  raise? 
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Pattern:  [Adverbial]  [NP]  [VP]. 

Examples:  In  December  terrorists  bombed  the  Bank  of  Colombia. 

On  December  15  assassins  kidnapped  the  Prime  Minister. 

In  Guatemala  City,  the  guerrillas  bombed  a  bank. 

Criteria:  Demonstrated  understanding  of  the  meaning  expressed  by  the  prepositional  phrase 
in  initial  position. 


Score: 


1.2  Adverbials  in  Pre-Verb  Position 

In  the  following  subsections,  we  will  test  adverbials  in  the  pre-verb  position,  that  is,  the 
position  between  the  subject  and  the  verb.  Below  are  examples  of  adverbials  in  pre-verb 
position.  The  adverbials  are  bracketed: 

Eg.  List  the  branch  which  [in  1990]  hired  John  Smith. 

Eg,  Who  [always]  works  overtime? 

Eg,  What  branch  [only]  has  women? 

Eg.  Terrorists  [on  December  15]  bombed  the  Bank  of  Colombia. 

Eg,  Assassins  [yesterday]  kidnapped  the  Prime  Minister. 


1.2.1  Simple  Adverbs  in  Pre-Verb  Position 


Use  any  sentence  pattern(s)  from  previous  sections  which  the  system  has  successfully  pro¬ 
cessed.  Insert  an  [Adverbial]  in  front  of  the  [Verb]  or  [VP]  in  the  sentence  pattern.  Some 
examples  are  shown  below.  Now  replace  the  [Adverbial]  in  the  sentence  pattern  with  a  simple 
adverb. 

Pattern:  List  [NP]  [Rel  Pronoun]  [Adverbial]  [VP]  ? 

Example:  List  the  department  which  only  has  3  employees. 


Pattern:  [WH-Word]  ([NP])  [Adverbial]  [VP]  ? 

Example:  What  branch  only  has  women? 


Pattern:  [NP] 

Examples:  Terrorists 
Assassins 
The  guerrillas 


[Adverbial]  [VP]. 

today  bombed  the  Bank  of  Colombia, 

yesterday  kidnapped  the  Prime  Minister, 

possibly  attacked  the  farm. 


Criteria:  Demonstrated  understanding  of  the  meaning  expressed  by  the  simple  adverb  in 
pre-verb  position. 

_  Score:  _ 
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1.2.2  Prepositional  Phrases  as  Adverbials  in  Pre-Verb  Position 

Use  any  sentence  pattern(s)  from  previous  sections  which  the  system  has  successfully  pro¬ 
cessed.  Insert  an  [Adverbial]  in  front  of  the  [Verb]  or  [VP]  in  the  sentence  pattern.  Some 
examples  axe  shown  below.  Now  replace  the  [Adverbial]  in  the  sentence  pattern  with  a  simple 
prepositional  phrase  ([PP]). 

Pattern:  List  [NP]  [Rel  Pronoun]  [Adverbial]  [VP]  ? 

Example:  List  the  department  which  in  1990  hired  women. 

Pattern:  [WH-Word]  ([NP])  [Adverbial]  [VP]  ? 

Example:  What  employees  in  1990  sold  CPUs? 

Pattern:  [NP]  [Adverbial]  [VP]. 

Examples:  Terrorists  on  December  15  bombed  the  Bank  of  Colombia. 

Assassins  in  Guatemala  kidnapped  the  Prime  Minister. 

Criteria:  Demonstrated  understanding  of  the  meaning  expressed  by  the  prepositional  phrase 
as  adverbial  in  pre-verb  position. 


Score: 


1.3  Adverbials  in  Final  Position 

In  the  following  subsections,  we  will  test  adverbials  in  the  final  position,  that  is,  at  the 
end  of  the  sentence.  Below  are  examples  of  adverbials  in  final  position.  The  adverbials  are 
bracketed: 

Eg,  List  the  department  which  hired  women  [in  1990] . 

Eg,  List  the  employees  who  were  hired  [recently] . 

Eg,  List  the  employees  who  received  a  raise  [last  year] . 

Eg,  Who  was  hired  [in  1989]  ? 

Eg,  Who  works  [in  New  York]? 

Eg,  What  employees  sold  CPUs  [in  1990]? 

Eg,  Terrorists  bombed  the  Bank  of  Columbia  [yesterday] . 

Eg,  Assassins  kidnapped  the  Prime  Minister  [in  Guatemala] . 


1.3.1  Simple  Adverbs  in  Final  Position 

Use  any  sentence  pattern(s)  from  previous  sections  which  the  system  has  successfully  pro¬ 
cessed.  Add  an  [Adverbial]  to  the  end  of  the  sentence  pattern.  Some  examples  are  shown 
below.  Now  replace  the  [Adverbial]  in  the  sentence  pattern  with  a  simple  adverb. 

Pattern:  List  [NP]  [Rel  Pronoun]  [VP]  [Adverbial], 

Example:  List  the  employees  who  were  hired  recently. 
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Pattern:  [WH-Word]  ([NP])  [VP]  [Adverbial]  ? 

Example:  Who  is  paid  weekly? 

Pattern:  [NP]  [VP]  [Adverbial]. 

Examples:  Terrorists  bombed  the  Bank  of  Columbia  yesterday. 

The  guerrillas  bombed  the  embassy  yesterday. 

Criteria:  Demonstrated  understanding  of  the  meaning  expressed  by  the  simple  adverb  in 
final  position. 


Score: 


1.3.2  Prepositional  Phrases  as  Adverbials  in  Final  Position 


Use  any  sentence  pattern(s)  from  previous  sections  which  the  system  has  successfully  pro¬ 
cessed.  Add  an  [Adverbial]  to  the  end  of  the  sentence  pattern.  Some  examples  are  shown 
below.  Now  replace  the  [Adverbial]  in  the  sentence  pattern  with  a  prepositional  phrase 

([pp]). 


Pattern: 

List 

[NP] 

[Rel  Pronoun] 

[VP] 

[Adverbial]. 

Examples: 

List 

the  department 

which 

hired  women 

in  1990. 

List 

the  employees 

who 

were  hired 

in  1990. 

Pattern:  [WH-Word] 

Examples:  Who 
What 


((NP])  [VP] 

was  hired 
employees  sold  CPUs 


[Adverbial]  ? 
in  1989? 
in  1990? 


Pattern: 

Examples: 


[NP] 

Terrorists 
Assassins 
The  guerrillas 
The  guerrillas 


[VF] 

bombed  the  Bank  of  Colombia 
kidnapped  the  Prime  Minister 
attacked  the  presidential  farm 
bombed  the  American  Embassy 


[Adverbial], 
on  December  15. 
in  Guatemala, 
near  Guatemala  City, 
last  year. 


Criteria:  Demonstrated  understanding  of  the  meaning  expressed  by  the  prepositional  phrase 
as  adverbial  in  final  position. 


Score: 
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V.  VERBS  AND  VERB  PHRASES 


1  Temporal  Aspect  of  Verb  Phrases 

In  this  section,  we  will  test  a  system’s  ability  to  understand  the  temporal  nature  of  different 
categories  of  verbs,  and  the  system’s  ability  to  understand  the  different  tenses  and  aspectual 
meanings. 

1.1  Stative  Verbs 

A  stative  verb  is  one  which  denotes  a  state,  that  is,  one  in  which  there  is  no  inherent  concept 
of  change  and  one  in  which  there  is  no  inherent  concept  of  a  beginning  or  an  end. 

1.1.1  Quality  (Inherent) 

These  verbs  denote  relatively  permanent  and  inalienable  properties  of  the  subject  being 
referred  to,  for  example,  “ to  be  bom  at  (a  certain  time)”  or  “to  be  tall”.  Inherent  stative 
verbs  may  not  be  used  with  a  progressive  aspect: 

*A  dog  is  being  a  mammal. 


1.1.2  State  (Non-Inherent) 

The  non-inherent  statives  include  verbs  which  denote  private  states  (“know”,  “believe ”, 
“feel”,  “dislike”)  and  others  which  describe  non-inherent  configurations  of  the  world,  such  as 
“contain”,  “belong ”,  or  “resemble”.  These  verbs,  in  general,  are  not  used  with  a  progressive 
aspect: 

*Sally  is  resembling  her  father. 

*1  am  believing  the  proposition  X . 

*The  box  is  containing  several  objects. 

but  compare: 

/  am  thinking  that  the  Lakers  won’t  win. 

My  tooth  is  aching  today. 

which  express  more  of  an  evolving  process. 

1.1. 2.1  Private  States 

1.1. 2. 1.1  Intellectual  States 

Verbs  of  intellectual  states,  such  as  “know”,  “think”,  “believe”,  “wonder”,  “suppose”  and  so 
on,  are  often  followed  by  a  nominal  clause. 
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1.1. 2.1.2  Emotional/ Attitudinal  States 

There  appear  to  be  two  classes  of  verbs  denoting  emotional  or  attitudinal  states.  One  class, 
including  verbs  such  as  “intend”,  “wish”,  “want”,  “disagree”,  is  followed  by  a  nominal  clause 
as  object,  while  the  other,  which  includes  “like”,  “dislike”,  “pity”,  “disagree  with”,  can  take 
a  simple  object. 

1.1. 2. 1.3  Perceptual  States 

Verbs  of  perception  (such  as  “see”,  “hear”,  “taste”,  ...)  use  the  modals  “can”  or  “could” 
to  refer  to  a  state  of  perception.  The  simple  present  or  past  expresses  an  act  of  perception 
which  has  a  defined  beginning  and  end.  Compare: 

I  heard  the  bell  ring.  [Event  seen  as  a  whole] 

with 

/  could  hear  the  bell  ring.  [Perception  continuing  over  a  period] 

1.1. 2. 1.4  Bodily  Sensations 

Verbs  in  this  set,  which  includes  “hurt”,  “ache”,  “itch”,  “feel  cold”,  have  approximately  the 
same  meaning  in  the  simple  present  as  in  the  present  progressive. 

1.2  Stance 

These  verbs,  including  “live”,  “sit”,  “stand”,  “lie”,  are  like  statives  in  that  their  use  with 
the  simple  present  and  past  denote  a  state  (  “He  lives  in  Massachusetts”)  but  they  are  unlike 
statives  in  that  they  readily  occur  with  a  progressive  aspect,  denoting  a  temporary  (and  thus 
changing)  state  of  affairs: 

He  is  living  in  Massachusetts. 


1.3  Dynamic  Verbs 

Dynamic  verbs  denote  events.  They  may  have  an  inherent  durational  quality  or  not,  that 
is,  they  may  take  a  longer  or  shorter  extent  of  time.  Events  that  are  virtually  instantaneous 
are  called  punctual.  Events  of  a  longer  duration  are  called  durational.  Dynamic  verbs  may 
be  conclusive  or  non-conclusive,  that  is,  they  may  have  an  implicit  ending  or  not. 

You  may  use  any  successful  sentence  pattern  from  previous  sections  or  one  of  the  sentence 
patterns  below.  These  sentence  patterns  will  be  used  to  test  dynamic  verbs  in  the  following 
subsections. 

Pattern:  List  [NP]  [Rel  Pronoun]  [Verb]  ([NP])  ([PP])- 

Examples:  List  the  department  which  hires  in  December. 

List  the  women  who  work  in  the  New  York  branch. 
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Pattern: 

[WH-Word] 

([NP]) 

[Verb] 

([NP])  «PP])  ? 

Examples: 

Who 

works 

in  New  York? 

Which 

department  sold 

the  most  CPUs  in  1990? 

Pattern: 

[NP] 

[Verb] 

([NP]) 

«ppp- 

Examples: 

Terrorists 

bombed 

the  Bank  of  Colombia. 

Assassins 

kidnapped 

the  Prime  Minister. 

1.3.1  Non- Conclusive  Durative  (Activities  and  Goings-On) 

Non-conclusive  durative  verbs  have  an  associated  time  extent  of  an  indefinite  nature,  that 
is,  there  is  no  implicit  end  to  the  activity  expressed  by  the  verb.  If  a  non-conclusive  durative 
verb  takes  an  agent  it  is  called  an  “activity”.  If  it  doesn’t  take  an  agent  it  is  called  a 
“going-on”.  A  segment  of  an  activity  or  going-on  is  an  event  of  the  same  type.  Examples 
of  non-conclusive  durative  verbs  are  “ rain ”,  “ shine ”,  “ play ”,  “run”,  “work”,  “manage  (a 
department)”,  “detain”,  terrorize”. 

In  the  following  example,  in  order  to  demonstrate  understanding  of  the  non-conclusive  dura¬ 
tive  verb  “work” ,  a  system  would  need  to  show  that  the  event  occurs  at  the  time  considered 
by  the  system  to  be  the  current  time,  with  the  temporal  endpoint  of  the  event  unknown: 

List  the  employees  who  work  in  Department  77. 

In  the  next  example,  in  order  to  demonstrate  understanding  of  the  non-conclusive  durative 
verb  “remain”,  a  system  would  need  to  show  that  the  event  occurs  at  the  time  considered 
to  be  the  system’s  current  time,  with  the  endpoint  of  the  event  unknown: 

Terrorists  remain  in  power  in  El  Salvador. 

Test  a  sentence  with  a  non-conclusive  durative  verb.  The  following  are  examples  of  such 
sentences. 

Eg,  List  the  employees  who  work  in  Department  77. 

Eg,  Who  supervises  the  New  York  office? 

Eg.  Terrorists  hold  the  Mayor  prisoner. 

Eg,  Terrorists  remain  in  power  in  El  Salvador. 

Criteria:  Demonstrated  understanding  of  the  indefinite  nature  of  the  time  extent  of  a  non- 
conclusive  durative  verb,  i.e.,  that  there  is  no  implicit  end  to  the  activity  expressed  by  the 
verb. 


Score:. 


1.3.2  Conclusive  Durative  (Accomplishments  and  Processes) 
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Conclusive  durative  verbs  have  an  associated  time  extent,  and  they  express  an  ongoing 
event  which  culminates  in  some  completion.  A  conclusive  durative  verb  which  takes  an 
agent  is  called  an  “accomplishment”.  One  which  doesn’t  take  an  agent  is  called  a  “process”. 
Examples  of  conclusive  durative  verbs  are  “ripen”,  “ improve ”,  “earn  ($40,000)”,  “sabotage”, 
“cleans  (an  area)”. 

With  certain  adjuncts  or  objects,  activities  or  goings-on  (covered  in  the  previous  subsection) 
may  become  conclusive,  and  thus  fall  into  the  categories  accomplishments  and  processes. 
For  instance,  although  “eat”  and  “run”  are  activities,  “eat  dinner”  and  “run  a  mile”  are 
accomplishments. 

Test  an  accomplishment  or  process. 

Eg,  Who  distributes  to  the  New  York  region? 

Eg,  List  everyone  who  earned  $25,000  in  1989. 

Eg,  The  Shining  Path  sabotaged  an  electric  facility. 

Eg,  The  guerrillas  attacked  the  Government  House. 

Eg,  The  Armed  men  burned  vehicles  in  the  streets. 

Criteria:  Demonstrated  understanding  that  a  conclusive  durative  verb  expresses  a  durational 
event  which  culminates  in  a  state  of  completion. 

- - - Score: _ 


1.3.3  Non-conclusive  Punctual  (Momentary  Acts  and  Events) 

Non-conclusive  punctual  verbs  denote  events  which  involve  a  change  of  state  and  an  almost 
immediate  return  to  the  original  state.  These  events  are  of  a  very  short  duration,  or  even 
instantaneous.  For  this  reason,  non-conclusive  punctual  verbs  are  called  “momentary”.  If 
a  momentary  verb  takes  an  agent  it  is  called  a  “momentary  act”.  If  it  takes  no  agent  it 
is  called  a  “momentary  event”.  Examples  of  non-conclusive  punctual  verbs  are  “sneeze”, 
“flash”,  “tap”. 

If  non-conclusive  punctual  verbs  are  used  in  contexts  where  it  is  clear  that  there  is  a  longer 
time  duration  over  which  the  event  occurs,  they  are  understood  to  express  a  repetition  of 
the  momentary  event.  For  example,  in  the  sentence  “the  light  flashed  for  about  an  hour”,  it 
is  understood  that  the  light  flashed  a  number  times,  rather  than  just  once. 

Certain  punctual  verbs  such  as  “kill”,  “hire”,  “fire”  are  non-conclusive  when  used  with  no 
object  or  with  a  non-specific  object  such  as  a  class  of  things  (specified  by  a  plural  count 
noun),  and  conclusive  when  used  with  a  specific  object  (see  Section  X  Reference  for  more 
information  on  specific  and  non-specific  reference).  Note  that  sentence  (A),  with  a  non¬ 
specific  object,  is  non-conclusive  since  the  event,  a  series  of  hiring  occurrences,  is  ongoing, 
while  sentence  (B),  with  a  specific  object,  expresses  a  conclusive  event: 


(A)  Department  12  hires  women 

(B)  Department  14  fired  John  Smith. 
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Test  a  momentary  act  or  event  using  a  context  where  there  is  a  longer  time  duration  over 
which  the  event  occurs.  The  system  should  recognize  that  the  momentary  act  or  event 
repeats  over  the  longer  time  duration. 

Eg,  Who  hires  in  department  12? 

Eg.  List  the  manager  in  department  17  who  fires. 

Eg,  ARENA  killed  in  December. 

Eg,  ARENA  killed  civilians  in  December. 

Criteria:  Demonstrated  understanding  that  a  non- conclusive  punctual  verb  expresses  a  con¬ 
tinuing  repetitive  event  when  used  in  a  context  with  a  longer  time  duration. 

_ -  Score: _ 


1.3.4  Conclusive  Punctual  (Achievements  and  Transitory  Events) 

Conclusive  punctual  verbs  involve  a  change  of  state  which  happens  more  or  less  instanta¬ 
neously.  The  verbs  in  this  class  are  called  “transitory”.  Transitory  verbs  which  take  an 
agent  (transitory  acts)  are  called  “achievements”.  A  “transitory  event”  is  one  which  takes 
no  agent.  Examples  of  achievements  and  transitory  events  are  “begin ”,  “ end ”,  “die”. 

There  is  a  set  of  punctual  verbs  including  “kill ”,  “hire ”,  “fire”  which  are  conclusive  when 
used  with  a  specific  object.  For  example,  “kill  John  Smith”  is  conclusive  because  the  object 
John  Smith  has  undergone  a  conclusive  state  change  (in  this  case  going  from  the  state  of 
being  alive  to  that  of  being  dead). 

Test  an  achievement  or  a  transitory  event. 

Eg,  Who  vas  hired  in  1990? 

Eg.  List  the  employees  vho  left  in  1990. 

Eg,  Armed  terrorists  kidnapped  the  Guatemalan  Prime  Minister. 

Eg,  ARENA  assassins  killed  the  Prime  Minister. 

Eg,  The  guerrillas  bombed  the  American  Embassy  yesterday. 

Criteria:  Demonstrated  understanding  that  a  conclusive  punctual  involves  a  change  of  state 
which  is  thought  of  as  happening  more  or  less  instantaneously. 

_ Score: _ 


1.4  Tense  /  Aspect 

In  the  following  section,  we  test  tense  and  aspect  of  verbs.  Choosing  a  verb  which  the 
system  successfully  understood,  you  may  choose  from  the  following  sentence  patterns,  or 
any  successful  sentence  pattern  from  a  previous  section,  to  test  verb  tense  and  aspect.  In 
this  section,  and  later  sections,  we  refer  to  a  verb  head  with  any  auxiliaries  as  a  “verb  group ”, 
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which  we  will  denote  “[Verb  Group]”.  In  each  subsection  we  will  test  a  different  verb  group. 
You  will  replace  the  [Verb  Group]  in  the  suggested  sentence  pattern  with  the  specified  verb 
group. 

Pattern:  List  [NP]  [Rel  Pronoun]  [Verb  Group]  ([NP])  ([PP]). 

Examples:  List  the  department  which  hired  women  in  1990. 

List  the  women  who  are  working  in  New  York. 

Pattern:  [WH-Word]  ([NP])  [Verb  Group]  ([NP])  ([PP])  ? 

Examples:  What  employees  had  sold  CPUs  in  1990? 

Who  works  in  department  17? 

Pattern:  [NP]  [Verb  Group]  ([NP])  ([PP]). 

Examples:  Terrorists  have  been  bombing  downtown  San  Salvador  since  December  10. 

Assassins  kidnapped  the  Prime  Minister  on  December  15. 

In  the  examples  in  the  following  subsections  the  verb  group  is  enclosed  in  brackets  for  ease 
of  identification.  For  a  more  detailed  description  of  tense  and  aspect,  see  the  Glossary. 

1.4.1  Simple  Tense 

1.4. 1.1  Simple  Present 

The  present  tense  is  indicated  by  the  unmarked  form  of  the  verb,  as  in  “work”,  “play”, 
“sing”,  “stretch”,  with  the  exception  of  the  third  person  singular  form,  marked  by  a  final 
“-s ”  or  “-es”,  as  in  “works”,  “plays”,  “sings”,  “stretches”. 

The  present  tense  indicates  that  the  event  occurs  at  the  current  time.  Often  the  event  is 
understood  as  habitual  during  a  time  period  which  includes  the  current  time.  In  the  following 
example  the  work  is  habitual: 

Mr  Smith  works  late  on  Tuesdays. 

The  present  tense  of  achievements  and  transitory  events  cannot  be  construed  as  habitual, 
as  these  acts  and  events  can  only  occur  once.  For  example,  given  the  standard  reading  of 
“dying”,  the  following  sentence  does  not  make  sense: 

*Mr  Smith  dies  on  Tuesdays. 

Replace  the  [Verb  Group]  in  your  chosen  sentence  pattern  with  a  verb  in  present  tense, 
denoted  by: 

[Verb  -f-PREsj 

Eg,  List  the  employees  vho  [vork]  in  department  77. 

Eg,  Who  [hires]  in  department  12? 

Eg,  Terrorists  [assassinate]  Prime  Ministers. 
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Eg,  Peasants  [burn]  the  Government  House  every  Spring. 

Eg,  The  URNG  [claim]  responsibility  for  the  attack  on  the 
presidential  farm. 

Criteria:  Demonstrated  understanding  that  the  event  expressed  by  the  verb  in  the  present 
tense  occurs  at  the  time  which  the  system  considers  to  be  the  current  time. 

— _ Score: _ 


1.4. 1.2  Simple  Past 

The  past  tense  is  most  often  indicated  by  the  addition  of  the  “-ed”  suffix  to  a  verb,  as  in 
“worked”,  “played ”.  Another  common  form  of  the  past  tense,  with  verbs  with  a  short  “i” 
vowel  sound  (  “sing”,  “sit”)  is  the  replacement  of  the  “i”  with  an  “a”,  as  in  “sang”,  or  “sat”. 
In  verbs  with  a  final  “t”,  there  is  often  no  difference  between  the  present  and  the  past  tense 
forms,  for  example,  “hit”  or  “put”. 

The  past  tense  indicates  that  an  event  occurred  in  the  past.  The  event  may  be  a  specific 
event,  for  example: 

Armed  men  attacked  a  police  patrol. 

The  past  tense  may  also  be  used  with  an  event  which  was  habitual  during  a  time  period  in 
the  past,  for  example,  in: 

Smith  worked  from  1981  to  1990. 

Replace  the  [Verb  Group]  in  your  chosen  sentence  pattern  with  a  verb  in  past  tense,  denoted 
by: 


[Verb  +past] 

Eg,  List  the  employees  vho  [worked]  in  1990. 

Eg,  Vho  [sold]  vacuum  cleaners  in  1989? 

Eg,  The  terrorists  [kidnapped]  the  Prime  Minister. 

Eg,  Armed  men  [attacked]  the  Government  House  in  Peru. 

Eg,  The  guerrillas  [killed]  a  civilian  on  February  4. 

Criteria:  Demonstrated  understanding  that  the  event  expressed  by  the  verb  in  the  past  tense 
occurred  at  a  time  preceding  the  system’s  current  time. 

- Score: _ 
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1.4. 1.3  Simple  Future 

The  future  tense  is  indicated  by  the  use  of  the  modal  operator  “will”  followed  by  the 
unmarked  verb.  The  future  tense  indicates  an  event  (which  may  be  of  a  habitual  or  an 
ongoing  nature)  which  will  take  place  in  the  future,  as  in: 

The  terrorists  will  kidnap  the  Prime  Minister  in  February. 

Replace  the  [Verb  Group]  in  your  chosen  sentence  pattern  with  a  verb  in  future  tense,  fitting 
the  pattern: 

will  [Verb] 

Eg.  List  the  women  who  [will  supervise]  in  1990. 

Eg,  Who  [will  work]  in  the  New  York  office  in  1991? 

Eg,  The  terrorists  [will  kidnap]  the  Prime  Minister  in  February. 
Eg,  Armed  men  [will  attack]  the  Government  House  in  Peru  tomorrow. 
Eg,  The  URNG  [will  claim]  responsibility  for  the  attack  on  the 
presidential  farm. 

Criteria:  Demonstrated  understanding  that  event  expressed  by  the  verb  in  the  future  tense 
will  occur  at  a  time  that  follows  or  succeeds  the  system’s  current  time. 

_ Score: _ 


1.4.2  Perfect 

The  perfect  aspect  is  marked  by  the  use  of  the  [HAVE- Verb]  preceding  a  [Verb  (-ed)].  The 
[HAVE- Verb]  is  tensed,  and  the  [Verb  (-ed)]  is  called  the  -ed  participle,  or  the  perfect 
participle.  (It  is  also  the  form  which  is  found  in  passive  constructions,  called  the  pas¬ 
sive  participle.)  It  is  the  verb  with  an  “-ed”  or  “-en”  ending,  though  English  has  many 
irregularities.  Examples  of  the  perfect  aspect  are: 


present  perfect 
have  worked 
has  arrested 
have  sold 


past  perfect 
had  hired 
had  kidnapped 
had  sung 


future  perfect 

will  have  fired 
will  have  attacked 
will  have  eaten 


The  unmarked  form  of  the  [HAVE- Verb]  is  “have”.  The  present  tense  form  is  “have”  or 
“ has ”,  with  “has”  being  used  with  a  third  person  singular  (  “he”,  “she”,  “it”)  subject.  The 
past  tense  form  is  “had”. 

When  the  perfect  aspect  is  used  with  a  stative  verb  or  a  non-conausive  durative  verb,  the 
event  expressed  by  the  verb  group: 
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•  begins  at  a  time  anterior  to  that  signaled  by  tense 

•  the  event  occurs  within  a  time  frame  leading  up  to  the  time  indicated  by  tense  and 
may  continue  through  the  time  indicated  by  tense. 

For  example,  compare  the  following  two  sentences: 

simple  past:  Smith  worked  here  for  10  years. 

present  perfect:  Smith  has  worked  here  for  10  years. 

The  event  described  using  the  simple  past  has  already  concluded,  while  the  event  described 
using  the  present  perfect  begins  before  the  present  but  has  not  necessarily  concluded  at  the 
present  time. 

With  a  conclusive  verb,  the  perfect  aspect  does  not  impart  the  same  open-endedness  as  with 
a  non-conclusive  verb,  for  example: 

The  cherries  have  ripened. 

The  project  has  begun. 

A  sentence  expressed  with  the  perfect  aspect  will  have  a  different  focus  than  the  same 
sentence  expressed  with  the  simple  aspect  (by  simple  aspect  we  mean  with  no  overt  aspect 
marking,  as  in  the  simple  tenses  in  the  previous  subsections).  The  sentence  with  the  simple 
aspect  will  have  as  the  focus  the  actual  event,  while  the  sentence  with  the  perfect  aspect  will 
have  as  the  focus  the  state  resulting  from  the  event.  For  example,  compare: 

simple  aspect:  I  met  your  sister. 

perfect  aspect:  I  have  met  your  sister. 

The  first  sentence,  with  no  aspectual  marking,  focuses  on  the  event  mentioned  (the  actual 
meeting),  while  the  second,  the  perfect,  focuses  on  the  state  resulting  from  the  event  (that 
I  already  know  your  sister). 

1.4. 2.1  Present  Perfect 

Because  of  the  differences  between  non-conclusive  durative  verbs  and  other  verbs  with  respect 
to  the  perfect  aspect,  in  this  section  we  will  test  the  perfect  aspect  first  with  a  non-conclusive 
durative  verb,  and  then  with  any  other  category  verb. 

1.4.2. 1.1  With  a  Non-conclusive  Durative  Verb  (Activities  or  Goings-On) 

When  the  perfect  aspect  is  used  with  a  non-conclusive  durative  verb,  there  is  an  open-ended 
quality  to  the  event  expressed  by  the  verb  group.  For  example,  in: 

Smith  has  worked  here  for  10  years. 
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there  is  the  understanding  that  the  habitual  event  of  Smith’s  working  is  not  concluded  at 
the  present  time. 

Using  an  activity  or  a  going-on,  replace  the  [Verb  Group]  in  your  chosen  sentence  pattern 
with  a  verb  expressed  with  the  present  perfect,  which  fits  the  pattern: 

have /has  [Verb  (-ed)] 

Eg,  List  the  women  who  [have  supervised]  a  department  in  1991. 

Eg,  Which  salespersons  [have  worked]  in  New  York? 

Eg,  Terrorists  [have  held]  the  Mayor  prisoner  since  September. 

Eg,  Terrorists  [have  remained]  in  power  in  El  Salvador  since  March. 


Criteria:  Demonstrated  understanding  that  the  present  perfect  used  with  a  non-conclusive 
durative  verb  indicates  that  the  event: 
e  begins  in  the  past 

•  continues  to  the  present  and  perhaps  into  the  future. 

_ _ _ _  Score: _ 


1.4. 2. 1.2  With  a  Conclusive  or  Punctual  Verb 

With  a  conclusive  verb,  the  perfect  aspect  does  not  impart  the  same  open-endedness  as  with 
a  non-conclusive  verb.  For  example,  in 

The  cherries  have  ripened. 

it  is  understood  that  the  process  is  complete;  the  cherries  are  ripe. 

Using  any  conclusive  verb  (accomplishments,  processes,  achievements,  or  transitory  event) 
or  a  momentary  act  or  event,  replace  the  [Verb  Group]  in  your  chosen  sentence  pattern  with 
a  verb  expressed  with  the  present  perfect,  which  fits  the  pattern: 

have/has  [Verb  (-ed)] 

Eg,  List  the  salespersons  who  [have  sold]  more  than  five  vacuum 
cleaners . 

Eg,  Which  employees  [have  earned]  $45,000? 

Eg,  The  terrorists  [have  kidnapped]  the  Prime  Minister. 

Eg,  Armed  men  [have  bombed]  the  Government  House  in  Peru. 

Eg,  The  guerrillas  [have  killed]  a  civilian. 

Eg,  The  guerrillas  [have  attacked]  the  presidential  farm. 
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Criteria:  Demonstrated  understanding  that  the  present  perfect  used  with  a  conclusive  or 
punctual  verb  indicates  that  the  event: 

•  begins  in  the  past 

•  has  come  to  a  conclusion  by  the  present  moment. 

_ Score: _ 


1.4. 2. 2  Pluperfect  (Past  Perfect) 

An  event  expressed  with  the  past  perfect  aspect  begins  before  some  temporal  reference  point 
in  the  past,  possiby  established  by  an  expression  within  the  current  sentence  or  a  previous 
sentence.  In  the  sentence: 

Laura  Jones  had  worked  in  department  12  before  John  Smith  was  hired. 

the  temporal  reference  point  in  the  past  is  the  point  at  which  John  Smith  was  hired,  and 
it  is  understood  that  Laura  Jones’  working  starts  before  Smith’s  working  begins,  and  may 
continue  up  until  and  even  beyond  Smith’s  hire  date. 

Using  any  verb,  replace  the  [Verb  Group]  in  your  chosen  sentence  pattern  with  a  verb  ex¬ 
pressed  with  the  past  perfect,  which  fits  the  pattern: 

had  [Verb  (-ed)] 

Eg,  Who  [had  worked]  in  department  12  before  John  Smith  was  hired? 
Eg,  List  the  women  who  [had  earned]  $40,000  by  1989. 

Eg,  The  terrorists  [had  kidnapped]  the  Prime  Minister. 

Eg,  Armed  men  [had  attacked]  the  Government  House  in  Peru. 

Eg,  The  guerrillas  [had  attacked]  the  presidential  farm  before. 

Eg,  The  guerrillas  [had  bombed]  the  bank  earlier. 

Criteria:  Demonstrated  understanding  that  the  past  perfect  indicates  that  the  event  begins 
before  some  temporal  reference  point  which  is  in  the  past  relative  to  the  system’s  current 
time. 


Score:. 


1.4. 2. 3  Future  Perfect 

An  event  expressed  with  the  future  perfect  aspect  begins  before  some  temporal  reference 
point  in  the  future,  possibly  established  by  an  expression  within  the  current  sentence  or  a 
previous  sentence.  In  the  sentence: 

Laura  Jones  will  have  worked  in  department  12  before  John  Smith  will  be  hired. 
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the  temporal  reference  point  in  the  future  is  the  point  at  which  John  Smith  will  be  hired , 
and  it  is  understood  that  Laura  Jones’  working  starts  before  Smith’s  hiring  date,  and  may 
continue  up  until  and  even  beyond  that  time. 

Using  any  verb,  replace  the  [Verb  Group]  in  your  chosen  sentence  pattern  with  a  verb  ex¬ 
pressed  with  the  future  perfect,  which  fits  the  pattern: 

will  have  [Verb  (-ed)] 


Eg,  Which  salespersons  [will  have  sold]  more  than  Smith  in  1991? 
Eg.  List  the  women  who  [will  have  earned]  $40,000  by  1991. 

Eg,  Armed  men  [will  have  bombed]  the  telephone  company  building 
by  May  1 . 

Eg,  The  terrorists  [will  have  kidnapped]  the  Prime  Minister. 

Eg,  By  tomorrow,  the  terrorists  [will  have  kidnapped]  the  Prime 
Minister. 


Criteria:  Demonstrated  understanding  that  the  future  perfect  indicates  that  the  event  begins 
before  some  temporal  reference  point  which  is  in  the  future  relative  to  the  system’s  current 
time. 

_ Score: _ 


1.4.3  Progressive 

The  progressive  aspect  is  marked  by  the  use  of  the  [BE- Verb]  preceding  a  [Verb  (-ing)].  The 
forms  of  the  [BE- Verb]  are  as  follows: 


present  tense: 

“am” 

(used  with  “/”) 

“is” 

(used  with  “he”) 

“are” 

(used  with  “we”,  “you”,  “they”) 

past  tense: 

“was” 

(used  with  “I”,  “he”) 

“were” 

(used  with  “we”,  “you”,  “they”) 

future  tense: 

“ will ”,  followed  by  the  unmarked  “be” 

Examples  of  the  progressive  aspect  are: 


present  prog. 
am  working 
is  earning 
are  attacking 


past  prog. 

was  working 
was  earning 
were  attacking 


future  prog. 
will  be  working 
will  be  earning 
will  be  attacking 
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When  the  progressive  aspect  is  used  in  a  verb  group,  it  imparts  the  following  meaning  to 
the  event  expressed  by  the  verb: 

•  not  being  bounded  in  time,  that  is,  there  is  no  implied  beginning  and  no  implied 
endpoint. 

•  the  event’s  lack  of  completion. 

•  the  event’s  occurrence  at  the  time  signaled  by  tense. 

For  example, 

Mr.  Smith  is  working  (now). 

The  cherries  are  ripening. 

The  progressive  aspect  lends  a  durative  quality  to  punctual  events.  For  example,  with 
momentary  acts  and  events,  which  do  not  specify  a  lasting  change  in  state,  the  progressive 
aspect  indicates  a  continuing  repetition  of  the  act  or  event: 

The  light  is  flashing. 

The  company  is  hiring  several  employees. 

With  achievements  or  transitory  events,  which  specify  an  almost  instantaneous  lasting  change 
in  state,  the  use  of  the  progressive  results  in  the  non- conclusive  part  of  the  act  or  event  being 
stretched  over  time: 

Mr.  Smith  is  hiring  John  Doe  right  now.  [John  Doe  is  not  yet  hired] 


1.4. 3.1  Present  Progressive 

Because  of  the  differences  found  among  the  several  categories  of  verbs  with  respect  to  the 
progressive  aspect,  in  this  section  we  will  test  the  progressive  aspect  with  a  durative  verb  (one 
of  either  an  activity,  going-on,  accomplishment,  or  process),  a  punctual  non-conclusive  verb 
(a  momentary  act  or  event),  and  a  punctual  conclusive  verb  (an  achievement  or  a  transitory 
event). 

1.4. 3. 1.1  With  a  Durative  Verb 

The  progressive  aspect  used  with  a  durative  verb  indicates  that  the  event  expressed  by  the 
verb  is  ongoing  at  the  current  time,  as  in: 

Mr.  Smith  is  working. 

The  cherries  are  ripening. 

Using  a  durative  verb  (either  an  activity,  going-on,  accomplishment,  or  process),  replace 
the  [Verb  Group]  in  your  chosen  sentence  pattern  with  a  verb  expressed  with  the  present 
progressive,  which  fits  the  pattern: 


V-13 


[BE- Verb  +pres]  [Verb  (-ing)] 


Eg,  Which  salespersons  [are  working]  in  New  York? 

Eg,  List  the  women  who  [are  supervising]  a  department. 

Eg,  The  arsonists  [are  burning]  the  Government  House. 

Eg,  Armed  men  [are  holding]  the  Prime  Minister  prisoner. 

Eg,  The  guerrillas  [are  conducting]  bombing  activities  in  the  city. 
Eg,  The  rebels  [are  burning]  vehicles  in  the  streets. 

Criteria:  Demonstrated  understanding  that  the  progressive  aspect  used  with  a  durative  verb 
indicates  that  the  event  has  no  implied  beginning,  and  is  ongoing  at  the  present  time  with 
no  implied  endpoint. 

_ _ _ Score: _ 


1.4. 3. 1.2  With  a  Punctual  Non- Conclusive  Verb  (Momentary  Act  or  Momentary  Event) 

The  progressive  aspect  lends  a  durative  quality  to  punctual  events.  For  example,  with 
momentary  acts  and  events,  which  do  not  specify  a  lasting  change  in  state,  the  progressive 
aspect  indicates  a  continuing  repetition  of  the  act  or  event. 

The  light  is  flashing. 

The  company  is  hiring  several  employees. 

Using  a  momentary  act  or  momentary  event,  replace  the  [Verb  Group]  in  your  chosen  sentence 
pattern  with  a  verb  expressed  with  the  present  progressive,  which  fits  the  pattern: 

[BE- Verb  +pres]  [Verb  (-ing)] 

Eg.  List  the  department  which  [is  hiring]  women. 

Eg,  Which  salespersons  [are  selling]  CPUs  in  New  York? 

Eg,  ARENA  [is  bombing]  banks  in  El  Salvador. 

Eg,  Armed  men  [are  killing]  civilians  in  Peru. 

Eg,  The  guerrillas  [are  robbing]  banks  in  Guatemala. 

Criteria:  Demonstrated  understanding  that  the  progressive  aspect  used  with  a  punctual  non- 
conclusive  verb  indicates  a  continuing  repetition  of  the  act  or  event  expressed  by  the  verb, 
with  no  implied  beginning. 

_ —  Score: _ 


1.4.3. 1.3  With  a  Punctual  Conclusive  Verb  (Achievement  or  Transitory  Event) 
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With  achievements  or  transitory  verbs,  which  express  an  almost  instantaneous  change  in 
state,  the  progressive  aspect  signals  that  the  preliminary  events  leading  up  to  the  change  of 
state  are  ongoing.  The  state  change  is  not  concluded.  Given  the  sentence 

The  Mayor  is  dying  from  gunshot  wounds. 

it  is  understand  that  the  Mayor  is  not  yet  dead. 

Using  an  achievement  or  transitory  event,  replace  the  [Verb  Group]  in  your  chosen  sentence 
pattern  with  a  verb  expressed  with  the  present  progressive,  which  fits  the  pattern: 

[BE- Verb  +pres]  [Verb  (-ing)] 

Eg,  List  the  department  which  [is  finishing]  a  project. 

Eg,  List  the  employee  who  [is  selling]  the  most. 

Eg,  The  Mayor  [is  dying]  from  wounds  from  the  terrorist  attack 
in  December. 

Eg,  Armed  men  [are  assassinating]  the  Prime  Minister. 

Eg,  The  guerrillas  [are  stealing]  food  from  the  presidential  farm. 

Criteria:  Demonstrated  understanding  that  the  progressive  aspect  used  with  a  punctual 
conclusive  verb  indicates  that  the  preliminary  events  leading  up  to  a  change  of  state,  which 
has  not  yet  occurred,  are  ongoing. 

_ Score: _ 


1.4. 3. 2  Past  Progressive 

The  past  progressive  indicates  that  the  event  expressed  by  the  verb  is  ongoing  and  not 
complete  at  some  time  in  the  past.  In  the  following  sentence,  the  attack  is  ongoing  at  the 
point  in  time  of  the  shooting  of  the  president: 

The  terrorists  were  attacking  the  Government  House,  when 
the  President  was  shot. 

Using  any  verb,  replace  the  [Verb  Group]  in  your  chosen  sentence  pattern  with  a  verb  ex¬ 
pressed  with  the  past  progressive,  which  fits  the  pattern: 

[BE- Verb  H-past]  [Verb  (-ing)] 

Eg,  Which  salespersons  [were  working]  in  New  York  in  1990? 

Eg,  List  the  woman  who  [was  supervising]  in  1989. 

Eg,  The  terrorists  [were  attacking]  the  Government  House. 
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Eg,  Armed  men  [were  bombing]  an  electric  facility. 

Eg,  The  guerrillas  [were  attacking]  the  presidential  farm  in 
February . 

Eg,  The  rebels  [vere  stealing]  food  this  morning. 


Criteria:  Demonstrated  understanding  that  the  past  progressive  aspect  indicates  that  the 
event  was  ongoing  and  not  concluded  at  some  time  in  the  past. 

_ Score: _ 


1. 4.3.3  Future  Progressive 

The  future  progressive  indicates  that  the  event  expressed  by  the  verb  is  ongoing  and  not 
complete  at  some  time  in  the  future.  In  the  following  sentence,  the  meeting  of  the  conference 
will  be  ongoing  at  the  time  of  Mr  Bush’s  flight: 

The  Latin  American  Peace  Conference  will  be  meeting  when 
Mr  Bush  flies  to  Guatemala. 

Using  any  verb,  replace  the  [Verb  Group]  in  your  chosen  sentence  pattern  with  a  verb  ex¬ 
pressed  with  the  future  progressive,  which  fits  the  pattern: 

will  be  [Verb  (-ing)j 


Eg.  List  the  women  who  [will  be  supervising]  a  department. 

Eg,  Which  salespersons  [will  be  working]  in  New  York  in  1992? 

Eg,  The  terrorists  [will  be  attacking]  the  Government  House. 

Eg,  Armed  men  [will  be  bombing]  am  electric  facility. 

Eg,  The  guerrillas  [will  be  attacking]  the  presidential  farm  next 
week. 

Eg,  The  guerrillas  [will  be  bombing]  the  banks  soon. 


Criteria:  Demonstrated  understanding  that  the  future  progressive  aspect  indicates  that  the 
event  is  ongoing  at  some  point  in  the  future. 

_ Score: _ 


1.4.4  Perfect  Progressive 

The  perfect  progressive  aspect  is  marked  by  the  use  of  the  [HAVE- Verb]  followed  by  the  -ed 
participle  of  the  [BE- Verb],  “ been ”,  followed  by  a  [Verb  (-ing)].  For  example, 


V-16 


present  perf.  prog. 
has  been  working 
have  been  eating 
has  been  arresting 


past  perf.  prog. 
have  been  hiring 
had  been  kidnapping 
had  been  kidnapping 


future  perf.  prog. 
had  been  firing 
will  have  been  attacking 
will  have  been  attacking 


The  unmarked  form  of  the  [HAVE- Verb]  is  “have”.  The  present  tense  form  is  “have ”  or 
“has”,  with  “has”  being  used  with  a  third  person  singular  (  “he”,  “she”,  “it”)  subject.  The 
past  tense  form  is  “had”. 

When  the  perfect  progressive  aspect  is  used  in  a  verb  group,  it  imparts  the  following  meaning 
to  the  event  expressed  by  the  verb: 

•  an  extent  in  time,  or  a  duration,  to  the  event  described. 

•  the  event  begins  before  the  reference  point,  the  time  indicated  by  tense. 

•  the  event’s  lack  of  completion  at  the  reference  point. 

Contrasting  the  present  progressive  with  the  present  perfect  progressive,  we  see  that  in  both 
cases  there  is  an  unboundedness  in  time  indicated  by  the  progressive,  and  we  can  see  that 
the  perfect  supplies  the  meaning  of  the  event  starting  at  an  earlier  point  in  the  past: 

present  progressive:  I  am  writing  a  paper. 

present  perfect  progressive:  I  have  been  writing  a  paper. 

The  incompleteness  of  an  event,  which  is  inherent  in  events  described  using  the  progressive 
aspect,  though  not  necessarily  inherent  in  events  described  with  the  perfect  aspect,  is  found 
with  the  perfect  progressive.  Compare  the  perfect  progressive,  above,  with  the  perfect: 

present  perfect:  I  have  written  a  paper. 

As  with  the  progressive,  there  is  an  durative  quality  given  to  punctual  events  which  are 
described  with  the  perfect  progressive.  With  momentary  acts  and  events,  which  specify  a 
change  in  state  and  an  almost  immediate  return  to  the  original  state,  the  perfect  progressive 
aspect  indicates  a  continued  repetition  of  the  act  or  event,  beginning  anterior  to  the  time 
indicated  by  tense: 

The  light  has  been  flashing. 

Mr.  Smith  has  been  hiring  for  two  years.  [In  charge  of  hiring] 

With  achievements  or  transitory  events,  which  specify  an  almost  instantaneous  lasting  change 
in  state,  the  use  of  the  perfect  progressive  results  in  the  non-conclusive  part  of  the  act  or 
event  being  stretched  over  time,  beginning  anterior  to  the  time  indicated  by  tense: 

The  terminally-ill  patient  has  been  dying  for  several  days. 
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1.4.4. 1  Present  Perfect  Progressive 

The  differences  found  among  the  several  categories  of  verbs  with  respect  to  the  perfect 
progressive  aspect  are  also  found  with  respect  to  the  progressive  aspect,  and  have  been 
tested  above,  in  that  section.  Therefore,  we  will  not  differentiate  among  the  different  classes 
of  verbs  here. 

The  present  perfect  progressive  aspect  indicates  that  the  event  has  a  time  duration  beginning 
in  the  past,  and  not  completed  at  present. 

Using  any  verb,  replace  the  [Verb  Group]  in  your  chosen  sentence  pattern  with  a  verb  ex¬ 
pressed  with  the  present  perfect  progressive,  which  fits  the  pattern: 

have/has  been  [Verb  (-ing)] 

Eg,  Which  salespersons  [have  been  working]  in  New  York? 

Eg,  List  the  PhD  who  [has  been  supervising]  a  department. 

Eg,  The  terrorists  [have  been  attacking]  the  Government  House. 

Eg,  Armed  men  [have  been  sabotage]  an  electric  facility. 

Eg,  The  guerrillas  [have  been  attacking]  the  farm  since  yesterday. 
Eg,  The  rebels  [have  been  burning]  vehicles  in  the  streets. 

Criteria:  Demonstrated  understanding  that  the  present  perfect  progressive  aspect  indicates: 

•  an  extent  in  time,  or  a  duration,  to  the  event  expressed  by  the  verb 

•  the  event  begins  in  the  past 

•  the  event  is  not  completed  at  present. 

_ Score: _ 


1.4. 4.2  Past  Perfect  Progressive 

The  present  perfect  progressive  aspect  indicates  that  the  event  has  a  time  duration  beginning 
prior  to  a  temporal  reference  point  in  the  past  and  was  not  completed  at  that  past  temporal 
reference  point. 

Using  any  verb,  replace  the  [Verb  Group]  in  your  chosen  sentence  pattern  with  a  verb  ex¬ 
pressed  with  the  past  perfect  progressive,  which  fits  the  pattern: 

had  been  [Verb  (-ing)] 

Eg.  List  the  women  who  [had  been  supervising]  a  department  in  1989. 
Eg,  Which  salespersons  [had  been  working]  in  New  York  in  1989? 

Eg,  The  terrorists  [had  been  attacking]  the  Government  House. 

Eg,  Armed  men  [had  been  sabotaging]  an  electric  facility. 

Eg,  The  guerrillas  [had  been  attacking]  the  presidential  farm  in 
1989. 
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Criteria:  Demonstrated  understanding  that  the  past  perfect  progressive  aspect  indicates: 

•  an  extent  in  time,  or  a  duration,  to  the  event  expressed  by  the  verb 

•  the  event  begins  prior  to  a  temporal  reference  point  in  the  past 

•  the  event  is  not  completed  by  the  past  temporal  reference  point. 

_ Score: _ 


1.4.4. 3  Future  Perfect  Progressive 

The  future  perfect  progressive  aspect  indicates  that  the  event  has  a  time  duration  beginning 
prior  to  a  temporal  reference  point  in  the  future  and  is  not  completed  at  that  future  temporal 
reference  point. 

Using  any  verb,  replace  the  [Verb  Group]  in  your  chosen  sentence  pattern  with  a  verb  ex¬ 
pressed  with  the  future  perfect  progressive,  which  fits  the  pattern: 

will  have  been  [Verb  (-ing)] 

Eg.  List  the  employees  who  [will  have  been  working]  12  years  in  1992. 
Eg,  The  terrorists  [will  have  been  attacking]  the  Government  House 
for  three  months  in  June. 

Eg,  Armed  men  [will  have  been  sabotaging]  an  electric  facility  for 
three  days. 

Eg,  The  guerrillas  [will  have  been  holding]  the  mayor  hostage  for 
a  week  today. 

Criteria:  Demonstrated  understanding  that  the  future  perfect  progressive  aspect  indicates: 

•  an  extent  in  time,  or  a  duration,  to  the  event  expressed  by  the  verb 

•  the  event  begins  prior  to  a  temporal  reference  point  in  the  future 

•  the  event  is  not  completed  by  the  future  temporal  reference  point. 

_ Score: _ 
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VI.  QUANTIFIERS 

Quantifiers  are  words  such  as  “all”,  “some”,  “many”  “none”,  and  “every”  which  indicate 
how  many  or  how  much  of  some  entity  is  being  referred  to.  Quantifiers  axe  found  in  a  number 
of  sentence  positions,  including  predeterminer  position,  central  determiner  position,  postde¬ 
terminer  position,  and  noun  phrase  head  position  (functioning  as  pronouns).  Remember  that 
the  determinative  element  of  a  noun  phrase  has  up  to  three  elements,  with  the  predeterminer 
being  first,  the  central  determiner  being  next,  and  the  postdeterminer  being  last.  For  more 
information  on  determinatives,  see  section  III,  Noun  Phrases ,  subsection  3,  Determinatives 
in  More  Detail.  In  the  following  examples  of  different  quantifier  position,  the  quantifier  is 
enclosed  in  brackets  for  ease  of  identification: 

Predeterminer  position: 

Here  are  [all]  the  employees  in  the  New  York  Branch. 


Central  determiner  position: 

[Some]  employees  recently  received  a  raise. 


Postdeterminer  position: 

The  [few]  employees  in  sales  also  received  promotions. 


Noun  phrase  head  position: 

[Five]  of  the  secretaries  received  mid-year  bonuses. 


In  the  first  subsection  we  test  quantifiers  in  certain  functional  positions.  In  the  second 
subsection  we  test  quantifiers  with  respect  to  meaning.  In  the  third  subsection  we  test 
quantifier  pronouns.  In  the  fourth  subsection  we  test  existential  there  sentences.  In  the  fifth 
subsection  we  test  universal  adjectives. 

1.  Functional  Position  of  Quantifiers 
1.1  Quantifiers  in  Determinative  Positions 

Quantifiers  which  occur  as  part  of  the  determinative  of  a  noun  phrase  are  referred  to  as 
determiner  quantifiers.  Quantifiers  can  occur  in  predeterminer,  central  determiner,  and 
postdeterminer  position. 

Below  are  some  suggested  sentence  patterns.  In  each  subsection,  you  will  replace  the  [NP 
+ det  quant]  (denoting  a  noun  phrase  with  a  quantifier  in  a  determiner  position)  with  a  noun 
phrase  which  incorporates  the  specified  type  of  determiner  quantifier. 

In  the  following  subsections  we  will  use  [NP  -det]  to  denote  a  noun  phrase  without  deter¬ 
minative.  In  the  examples  in  each  subsection  we  will  bracket  the  complete  noun  phrase 
including  the  quantifier  determinative  element. 
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Note  that  a  noun  phrase  includes  optional  postmodification. 

MAIN  DETERMINER  QUANTIFIER  PATTERNS: 

Pattern:  List  [NP  +det  quant]. 

Example:  List  all  employees  in  Department  77. 

Pattern:  List  [NP]  [Rel  Pronoun]  [Verb]  [NP  +det  quant]. 

Example:  List  the  salespersons  who  sold  five  vacuum  cleaners. 

Pattern:  What  ([NP])  [BE- Verb]  [NP  +det  quant]  [Verb  (-ing)]  ? 

Example:  What  items  are  all  the  six  salesmen  in  New  York  selling? 

Pattern:  What  ([NP])  [BE- Verb]  [NP  +  det  quant]  [Verb  (-ed)]  ? 

Example:  What  are  all  the  employees  in  payroll  paid? 

Pattern:  Who  [BE- Verb]  [NP  -f det  quant]  ? 

Example:  Who  are  both  the  Vice  Presidents  of  Shipping  ? 

Pattern:  What  ([NP])  [DO- Verb]  [NP  +det  quant]  [Verb]  ? 

Example:  What  benefits  do  some  employees  receive? 

Pattern:  What  ([NP])  [Verb]  [NP  -f- det  quant]  ? 

Example:  What  regions  sell  all  the  items  in  the  catalog  ? 

Pattern:  [NP  -(-det  quant]  [VP], 

Example:  Six  electric  facilities  in  Bolivia  were  bombed  by  terrorists. 

Pattern:  [NP]  [Verb]  [NP  4-det  quant]. 

Example:  The  terrorists  shot  the  few  Ambassadors  at  the  Peruvian  Embassy. 

1.1.1  Quantifiers  in  Predeterminer  Position 

Predeterminer  quantifiers  include  “all”,  “both”,  and  the  fractions  and  multipliers.  These 
quantifiers  are  called  “predeterminer  quantifiers”  because  they  are  used  to  precede  central 
determiners  (e.g.,  “the”,  “a/an”,  “this”,  “that”,  “these”,  “those”). 

1.1. 1.1  With  Count  Nouns 

Test  a  predeterminer  quantifier  with  a  count  noun.  Use  the  article  “the”  as  the  central 
determiner.  Use  “both”  only  when  quantifying  over  a  noun  phrase  which  expresses  a  group 
with  exactly  two  elements  in  it. 

Replace  the  [NP  +det  quant]  in  the  MAIN  DETERMINER  QUANTIFIER  PATTERNS 
with: 

Pattern:  [Predeterminer  Quantifier]  the  [NP  -det] 

Example:  all  the  employees  in  department  12. 

The  following  full  sentence  examples  incorporate  this  pattern: 

Eg,  List  [all  the  employees  in  department  12] . 
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Eg,  List  [twice  the  salary  of  John  Smith] . 

Eg,  What  are  [all  the  employees  in  the  payroll  department]  paid? 

Eg,  What  products  do  [both  the  women  in  Chicago]  sell? 

Eg,  What  is  [half  the  salary  of  John  Smith]? 

Eg,  The  terrorists  shot  [all  the  Ambassadors  at  the  Peruvian 
Embassy] . 

Eg,  Terrorists  bombed  [both  the  Telephone  Communications  buildings 
in  Nicaragua] . 

Eg,  Terrorists  killed  [half  the  border  patrol  at  the  Nicaraguan 
border] . 

Eg,  (Preliminary  Sentence:  Terrorists  attacked  twelve  border  guards 
at  the  Nicaraguan  border.)  The  terrorists  killed  [half  the 
guards] . 

Eg,  [All  the  electric  facilities  in  Bolivia]  were  bombed  by 
terrorists. 

Eg,  Two  Mayors  were  killed.  [Twice  the  number]  were  kidnapped. 

Eg,  [All  the  buildings]  were  burned  in  the  guerrilla  attack. 

Criteria:  Demonstrated  understanding  of  the  quantity  indicated  by  the  predeterminer  quan¬ 
tifier. 

_ Score: _ 


1.1. 1.2  With  Mass  Nouns 

Use  “all”  to  test  a  predeterminer  quantifier  with  a  mass  noun.  Use  the  article  “the”  as  the 
central  determiner. 

Replace  the  [NP  +det  quant]  in  the  MAIN  DETERMINER  QUANTIFIER  PATTERNS 
with: 

Pattern:  all  the  [NP  -DET,  +MASs] 

Example:  all  the  information  about  John  Smith. 

The  following  full  sentence  examples  incorporate  this  pattern: 

Eg,  List  [all  the  information  about  John  Smith]. 

Eg,  What  department  has  [all  the  computer  expertise]? 

Eg,  [All  the  information  about  ARENA]  was  destroyed  in  an  attack  on 
the  government  house. 

Eg,  [All  the  dynamite]  was  stolen  by  the  URNG. 


Criteria:  Demonstrated  understanding  of  the  totality  indicated  by  the  predeterminer  quan¬ 
tifier  “all”. 


Score: 
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1.1.2  Quantifiers  in  Postdeterminer  Position 

Postdeterminer  quantifiers  include  “many”,  “few”,  and  the  cardinal  numbers.  These 
quantifiers  are  called  “postdeterminer  quantifiers”  because  they  are  used  to  follow  a  central 
determiner  (e.g.,  “ the ”,  “a/an”,  “this”,  “that”,  “ these ”,  “those”). 

When  a  quantifier  follows  the  definite  article  “the”,  the  speaker  presupposes  the  quantity 
mentioned  is  correct.  For  example,  the  sentence 

List  the  one  research  assistant  in  department  17. 

is  an  incorrect  sentence  if  there  is  not  exactly  one  research  assistant  in  department  17  (that 
is,  if  there  are  zero,  or  two  or  more  research  assistants).  The  sentence 

List  the  many  managers  of  the  company. 

presupposes  that  there  are  many  managers.  If  there  are  not  many,  but  just  a  few,  the  sentence 
is  incorrect. 

Incorrect  use  of  quantifiers  as  described  above  is  a  form  of  “failed  presupposition”.  Do  not 
test  sentences  which  include  a  failed  presupposition  unless  you  are  specifically  requested  to 
do  so. 


1.1. 2.1  Successful  Presupposition 

Test  a  sentence  with  a  postdeterminer  quantifier  following  the  article  “the”.  Make  sure  there 
is  no  error  in  the  presupposition. 

Replace  the  [NP  +det  quant]  in  the  MAIN  DETERMINER  QUANTIFIER  PATTERNS 
with: 

Pattern:  the  [Postdeterminer  Quantifier]  [NP  -det] 

Example:  the  many  employees  in  department  7 

The  following  full  sentence  examples  incorporate  this  pattern: 

Eg,  List  [the  many  employees  in  department  7] . 

Eg,  List  [the  two  vomen  at  the  Chicago  branch] . 

Eg,  tfhat  do  [the  three  salespersons  in  the  New  York  office]  earn? 
Eg,  ARENA  kidnapped  [the  two  men  who  guard  the  Government  House] . 
Eg.  [The  few  terrorists]  attacked  the  Government  House. 

Eg,  [The  one  terrorist  who  kidnapped  the  Prime  Minister]  died  later 
in  an  explosion. 

Eg,  [The  two  presidential  farms]  were  attacked  by  the  URNG. 

Criteria:  Demonstrated  understanding  of  the  quantity  indicated  by  the  postdeterminer  quan¬ 
tifier. 
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Score: 


1.1. 2. 2  Failed  Presupposition 

Test  a  sentence  with  a  postdeterminer  quantifier  following  the  article  “the”.  Make  sure  there 
is  an  error  in  the  presupposition.  For  example,  if  there  are  only  five  salespersons  you  could 
say  “ List  the  seven  salespersons.  ” 

Replace  the  [NP  +det  quant]  in  the  MAIN  DETERMINER  QUANTIFIER  PATTERNS 
with: 

Pattern:  the  [Postdeterminer  Quantifier]  (NP  -det] 

Example:  the  many  employees  in  department  7 

The  following  full  sentence  examples  incorporate  this  pattern: 

Eg.  List  [the  two  women  at  the  Chicago  branch] . 

Eg.  List  [the  five  employees  who  earned  $40,000  in  1990]. 

Eg,  What  do  [the  three  salespersons  in  the  New  York  office]  earn? 

Eg,  (Preliminary  Sentence:  ARENA  kidnapped  two  officials.)  [The  many 
officials]  were  later  killed. 

Eg,  (Preliminary  Sentence:  Seven  terrorists  attacked  the  Government 
House.)  [The  eight  terrorists]  were  killed. 

Eg,  (Preliminary  Sentence:  Two  terrorists  kidnapped  the  Prime 

Minister.)  [The  one  terrorist  who  kidnapped  the  Prime  Minister] 
died  later  in  an  explosion. 

Criteria:  Demonstrated  understanding  that  an  error  occurred  because  the  amount  expressed 
by  the  postdeterminer  quantifier  following  the  definite  article  “the”  differed  from  the  known 
amount  of  the  entity  expressed  by  the  noun  phrase  being  quantified. 

- - - - - - Score: _ 


1.1.3  Quantifiers  in  Central  Determiner  Position 

Central  determiner  quantifiers  include  “each”,  “every”,  “some”,  “either”,  “any”,  “ nei¬ 
ther ”,  “no”,  and  so  on.  They  are  called  “central  determiner  quantifiers”  because  they  neither 
follow  nor  precede  a  central  determiner  (e.g.,  “the”,  “a/an”,  “this”,  “that”,  “these”,  “those”). 

Replace  the  [NP  +det  quant]  in  the  MAIN  DETERMINER  QUANTIFIER  PATTERNS 
with: 

Pattern:  [Central  Determiner  Quantifier]  [NP  -det] 

Example:  every  Manager  who  earns  above  $45,000 

The  following  full  sentence  examples  incorporate  this  pattern: 

Eg,  List  [every  Manager  who  earns  above  $45,000]. 
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Eg,  List  [some  employees  in  the  Chicago  branch] . 

Eg,  List  the  employees  vith  [no  degree] . 

Eg,  What  does  [each  employee]  earn? 

Eg,  What  products  does  [every  salesperson]  sell? 

Eg,  What  education  does  [either  voman  in  Neu  York]  have? 

Eg,  What  department  includes  [no  women]? 

Eg,  Ten  terrorists  bombed  electric  facilities.  [Every  terrorist] 
bombed  one  electric  facility. 

Eg,  [Some  information  about  ARENA]  was  destroyed  in  an  attack  on 
the  government  house. 

Eg,  [Neither  Marxist  group]  claimed  they  bombed  the  Government  House. 
Eg,  Terrorists  killed  [any  government  official  whom  they  saw] . 

Eg,  Terrorists  killed  [every  Senator]  in  the  attack  on  the 
Government  House. 

Criteria:  Demonstrated  understanding  of  the  quantity  indicated  by  the  central  determiner 
quantifier. 

_ _ _ _ _ Score: _ 


1.1.4  Predeterminer  and  Postdeterminer  Combinations 
1.1. 4.1  Without  a  Central  Determiner 

Replace  the  [NP  +det  quant]  in  the  MAIN  DETERMINER  QUANTIFIER  PATTERNS 
with: 

Pattern:  [Predeterminer  Quantifier]  [Postdeterminer  Quantifier]  [NP  -det] 

Example:  all  five  employees  in  Chicago 

The  following  full  sentence  examples  incorporate  this  pattern: 

Eg,  Who  are  [all  five  employees  in  Chicago]? 

Eg,  What  are  [both  two  women  in  Chicago]  paid? 

Eg,  List  [all  few  women  in  the  New  York  branch] . 

Eg,  [All  100  men  in  ARENA]  bombed  the  Government  House. 

Eg,  [All  few  Government  Officials]  were  killed. 

Eg,  ARENA  bombed  [all  seven  electric  facilities] . 

Eg,  ARENA  bombed  [all  few  electric  facilities] . 

Eg,  (Preliminary  Sentence:  Five  Farms  were  attacked  by  the  URNG.) 
[All  five  farms]  were  destroyed. 

Criteria:  Demonstrated  that  it  determined  the  correct  amount  expressed  by  the  combination 
of  predeterminer  and  postdeterminer  quantifiers,  and  correctly  interpreted  the  quantified 
noun  phrase. 

- - - -  Score:  _ 
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1.1. 4. 2  With  an  Article  as  Central  Determiner 


Use  the  article  “the”  as  the  central  determiner.  Replace  the  [NP  +det  quant]  in  the  MAIN 
DETERMINER  QUANTIFIER  PATTERNS  with: 

Pattern:  [Predeterminer  Quant.]  the  [Postdeterminer  Quant.]  [NP  -det] 

Example:  both  the  two  women 

The  following  full  sentence  examples  incorporate  this  pattern: 

Eg,  List  [half  the  many  employees  in  the  company] . 

Eg,  What  are  [both  the  two  women  in  Chicago]  paid? 

Eg,  Who  axe  [all  the  five  employees  in  Chicago]? 

Eg,  List  [all  the  few  women  in  the  New  York  branch] . 

Eg,  [All  the  100  men  in  ARENA]  bombed  the  Government  House. 

Eg,  [All  the  seven  Government  Officials]  were  killed. 

Eg,  ARENA  bombed  [both  the  electric  facilities] . 

Eg,  ARENA  bombed  [half  the  six  electric  facilities] . 

Eg,  [Both  the  two  presidential  farms]  were  attacked  by  the  URNG. 

Criteria:  Demonstrated  that  it  determined  the  correct  amount  expressed  by  the  combination 
of  predeterminer  and  postdeterminer  quantifiers  with  a  central  determiner,  and  correctly 
interpreted  the  quantified  noun  phrase. 

_ Score:  _ 


1.2  Quantifiers  as  Noun  Phrase  Head 

In  this  section  we  test  quantifiers  as  noun  phrase  head  with  a  partitive  o/-phrase.  A  par¬ 
titive  o/-phrase  is  structurally  a  prepositional  phrase  with  the  preposition  of.  A  quantifier 
expresses  an  amount  of  some  entity  and  the  noun  phrase  which  is  the  object  of  the  partitive 
o/prepositional  phrase  specifies  the  entity  being  quantified.  In  the  sentence  below,  the  entity 
is  the  collection  of  employees  in  Chicago  and  the  amount  being  expressed  is  all : 

All  of  the  employees  in  Chicago  are  well-paid. 

Quantifiers  used  as  heads  of  noun  phrases  are  called  “quantifier  pronouns”.  Quantifier  pro¬ 
nouns  include  “all”,  “both”,  “each”,  “some”,  “much”,  “many”,  “more”,  “most”,  “little”, 
“few”,  “fewer”,  “fewest”,  “less”,  “least”,  “any”,  “either”,  “none”,  “neither”,  and  the  car¬ 
dinal  numbers. 

Eg.  List  [all  of  the  employees  in  Chicago] . 

Eg,  List  [both  of  the  women  in  the  Chicago  branch] . 
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Eg.  List  the  positions  with  [many  of  the  benefits] . 

Eg.  List  the  department  with  [the  fevest  of  the  employees] . 

Eg,  List  [any  of  the  branches] . 

Eg,  List  [one  of  the  women  in  the  New  York  branch]. 

Eg,  What  is  [each  of  the  employees]  paid? 

Eg,  What  is  [some  of  the  information  about  Joe  Smith]? 

Eg,  What  departments  have  [more  of  the  PhDs]  ? 

Eg,  Are  [most  of  the  employees]  paid  over  $25,000? 

Eg,  Who  earns  [the  least  of  the  employees]? 

Eg,  [Few  of  the  Cabinet  Members]  were  killed  in  the  explosion  at 
the  Government  House. 

Eg,  [Each  of  the  three  Cabinet  Members]  vas  killed  by  a  carbomb. 

Eg,  Terrorists  burned  [much  of  the  information]  in  the  attack  on  the 
Government  House. 

Eg,  [Neither  of  the  Cabinet  Members]  were  killed  by  the  carbomb. 

Eg.  Terrorists  killed  [six  of  the  officials]  in  the  bombing  of 
the  Government  House. 

Eg,  [All  of  the  workers]  were  injured  in  the  URNG  attack  on  the 
Santo  Tomas  farm. 

Criteria:  Demonstrated  understanding  of  the  quantity  indicated  by  the  quantifier  pronoun. 

_ Score: _ 


2.  Semantics  of  Quantifiers 

In  the  following  subsections,  we  will  test  quantifiers  with  respect  to  their  meanings.  In  the 
previous  subsection,  we  tested  quantifiers  in  a  number  of  functional  positions,  including  pre¬ 
determiner  position,  central  determiner  position,  postdeterminer  position,  and  noun  phrase 
head  position.  In  the  following  subsections,  use  each  quantifier  in  a  position  which  has  been 
successfully  tested.  Do  not  use  a  sentence  which  has  a  failed  presupposition  unless  you  are 
specifically  requested  to  do  so. 

In  the  examples  in  this  subsection,  the  noun  phrase  in  which  the  quantifier  appears  either 
as  a  determinative  or  a  head  is  enclosed  in  brackets  for  ease  of  identification. 

2.1  Universal  Quantifiers 

2.1.1  ALL 

The  universal  quantifier  “all”  expresses  the  entire  quantity  or  amount;  totality. 

Eg,  List  [all  employees  in  Department  77] . 

Eg,  What  are  [all  employees  in  the  payroll  department]  paid? 

Eg,  What  benefits  do  [all  of  the  employees]  receive? 

Eg,  What  branches  have  [all  the  women]? 
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Eg,  [All  electric  facilities  in  -Bolivia]  were  bombed  by  terrorists. 
Eg,  [All  the  information  about  ARENA]  was  destroyed  in  an  attack  on 
Eg,  A  carbomb  killed  [all  of  the  Cabinet  Members] . 

Eg,  [All  the  buildings  at  the  presidential  farm]  vere  burned  in  a 
guerrilla  attack. 

Eg,  [All  of  the  workers]  vere  injured  in  an  URNG  attack  on  the  Santo 
Tomas  farm. 


Criteria:  Demonstrated  understanding  that  the  universal  quantifier  “ all ”  expresses  the  entire 
quantity  or  amount;  totality. 

_ Score: _ 


2.1.2  BOTH 


The  universal  quantifier  “both”  expresses  each  of  two. 

In  a  sentence  using  “both”  as  a  quantifier,  it  is  presupposed  that  the  quantified  noun  phrase 
expresses  exactly  two  objects.  For  example,  the  sentence 


List  both  research  assistants  in  department  17. 


is  in  error  if  there  are  not  exactly  two  research  assistants  in  department  17.  This  type  of 
error  is  known  as  failed  presupposition. 


2. 1.2.1  Successful  Presupposition 

Successful  presupposition  occurs  in  sentences  where  there  is  a  presupposition  which  is  not 
mistaken.  For  example,  in  the  sentence 

List  both  research  assistants  in  department  17. 

it  is  presupposed  that  there  are  exactly  two  research  assistants.  If  the  presupposition  holds, 
it  is  called  a  “successful  presupposition”. 

Test  “both”  as  a  quantifier  in  a  noun  phrase  which  refers  to  a  set  containing  exactly  two 
elements. 

Replace  the  [NP  +det  quant]  in  the  MAIN  DETERMINER  QUANTIFIER  PATTERNS 
with: 


both  (the)  [NP]:  (where  the  noun  phrase  expresses  two  objects) 

Eg,  List  [both  of  the  women]  in  department  12. 

Eg,  What  salaries  do  [both  the  Smiths]  earn? 

Eg,  [Both  the  Government  houses]  exploded. 

Eg,  [Both  of  the  Cabinet  Members]  were  killed  by  a  carbomb. 


VI-9 


Eg,  Terrorists  bombed  [both  Telephone  Communications  buildings]  in 
Nicaragua. 

Eg,  (Preliminary  Sentence:  Tvo  men  were  kidnapped  by  URNG 
guerrillas.)  [Both  the  kidnap  victims]  were  murdered  in 
Guatemala  City. 

Eg,  [Both  the  tvo  presidential  farms]  were  attacked  by  the  URNG. 


Criteria:  Demonstrated  understanding  that  the  universal  quantifier  “both”  expresses  each  of 
two. 

_ Score: _ 


2. 1.2.2  Failed  Presupposition 

Test  “both”  as  a  quantifier  for  a  noun  phrase  which  refers  to  a  set  that  does  not  contain  two 
elements.  The  system  should  give  evidence  that  it  recognizes  that  an  error  has  been  made. 

Replace  the  [NP  +det  quant]  in  the  MAIN  DETERMINER  QUANTIFIER  PATTERNS 
with: 


both  (the)  [NP]:  (where  the  number  of  the  [NP]  is  not  two) 

Eg,  List  [both  the  vomen  in  Chicago]  .  (where  there  is  only  one) 

Eg,  What  are  [both  of  the  salespersons]  paid?  (where  there  are  more  than  two) 
Eg,  Terrorists  kidnapped  the  Prime  Minister 
and  killed  [both  of  them] . 

Eg,  (Preliminary  Sentence:  Three  men  were  kidnapped  by  URNG 
guerrillas.)  [Both  the  kidnap  victims]  were  murdered  in 
Guatemala  City. 

Criteria:  Demonstrated  recognition  of  the  contradiction  between  the  quantifier  “both”  and 
the  number  of  objects  contained  in  the  set  expressed  by  the  noun  phrase  which  it  quantifies. 

_ Score: _ 


2.1.3  EACH 

The  universal  quantifier  “each”  expresses  every  one  of  the  distinct  individuals  (of  a  set 
containing  two  or  more  elements)  considered  separately  from  the  rest. 

“Each”  has  a  distributive  meaning.  That  is,  it  picks  out  the  members  of  a  set  singly,  rather 
than  as  a  mass.  Thus,  the  sentence: 

What  does  each  Manager  earn? 
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is  interpreted  as: 


Consider  the  managers  one  at  a  time.  What  does  the  manager  presently  being 
considered  earn? 


Eg,  What  does  [each  employee]  earn? 

Eg,  What  is  [each  of  the  women]  paid? 

Eg,  (Preliminary  Sentence:  Ten  terrorists  bombed  electric 

facilities.)  [Each  terrorist]  bombed  one  electric  facility. 

Eg,  Terrorists  killed  [each  of  the  five  union  leaders  in  Cuilapa] . 
Eg,  (Preliminary  Sentence:  Five  farms  were  attacked  by  URNG 
guerrillas.)  [Each  farm]  was  destroyed  in  the  attack. 

Criteria:  Demonstrated  understanding  that  the  universal  quantifier  * each "  expresses  every 
one  of  the  distinct  individuals  (of  a  set  containing  two  or  more  elements)  considered  sepa¬ 
rately  from  the  rest. 

— - _ Score: _ 


2.1.4  EVERY 

The  universal  quantifier  “every”  expresses  each  individual  or  part  of  a  group  without  excep¬ 
tion. 

“Every”  has  both  a  distributive  meaning  and  a  non-distributive  meaning.  If  its  meaning  is 
distributive,  it  picks  out  the  members  of  the  group  expressed  by  the  quantified  noun  phrase 
individually.  If  its  meaning  is  non-distributive,  it  treats  the  group  expressed  by  the  quantified 
noun  phrase  as  a  unit. 

Eg,  What  product  does  [every  employee]  sell? 

Eg,  List  [every  Manager  who  earns  above  $45,000]? 

Eg,  (Preliminary  Sentence:  Ten  terrorists  bombed  electric 

facilities.)  [Every  terrorist]  bombed  one  electric  facility. 

Eg,  (Preliminary  Sentence:  Ten  electric  facilities  were  bombed 
last  week.)  ARENA  had  bombed  [every  electric  facility  in  San 
Salvador] . 

Eg,  (Preliminary  Sentence:  Five  farms  were  attacked  by  URNG 
guerrillas.)  [Every  farm]  was  destroyed  in  the  attack. 

Criteria:  Demonstrated  understanding  that  the  universal  quantifier  “every”  expresses  each 
individual  or  part  of  a  group  without  exception. 

. — - _ Score: _ 
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2.2  Existential  Assertive  Quantifiers 
2.2.1  SOME 

The  existential  quantifier  “some”  expresses  a  part  of  an  object  or  an  unspecified  number  of 
objects. 


Eg,  List  [some  employees  in  the  Nev  York  office] . 

Eg,  What  products  do  [some  of  the  salespersons]  sell? 

Eg,  [Some  of  the  Cabinet  Members]  were  killed  by  a  carbomb. 

Eg,  Terrorists  killed  [some  peasants  in  Cuilapa] . 

Eg,  (Preliminary  Sentence:  Five  farms  were  attacked  by  URNG 
guerrillas.)  [Some  farms]  were  destroyed  in  the  attack. 

Eg,  [Some  dynamite]  vas  stolen  by  URNG  guerrillas  on  February  5. 

Criteria:  Demonstrated  understanding  that  the  existential  quantifier  “some”  expresses  a 
part  of  an  object  or  an  unspecified  number  of  objects. 

_ Score: _ 


2.2.2  SEVERAL 

The  existential  quantifier  “several”  expresses  more  than  two  but  fewer  than  many. 

Eg.  List  [several  employees] . 

Eg,  What  salary  do  [several  of  the  salespersons]  earn? 

Eg,  [Several  of  the  terrorists  in  ARENA]  attacked  the  Government 
House. 

Eg,  ARENA  kidnapped  [several  Government  Officials] . 

Criteria:  Demonstrated  understanding  that  the  existential  quantifier  “several”  expresses 
more  than  two  but  fewer  than  many. 

— - - Score: _ 


2.2.3  Multal 

“ MultaV  means  a  large  amount. 

2.2.3. 1  MANY 

The  existential  quantifier  “many”  expresses  a  large  but  indefinite  number. 


Eg.  List  [the  many  employees  in  department  7] . 
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Eg,  What  product  do  [many  salespersons]  sell? 

Eg,  What  degree  do  [many  of  the  employees]  have? 

Eg,  [Many  Cabinet  Members]  vere  killed  by  a  carbomb. 

Eg,  Terrorists  killed  [many  of  the  peasants  in  Cuilapa] . 

Eg,  [Many  peasants]  vere  injured  during  an  attack  by  URNG  guerrillas. 
Eg,  [Many  of  the  dynamite  sticks  at  the  presidential  farm]  vere 
stolen  by  guerrillas. 

Criteria:  Demonstrated  understanding  that  the  existential  quantifier  “many”  expresses  a 
large  but  indefinite  number. 

_  Score: _ 


2.2.3.2  MUCH 

The  existential  quantifier  “much”  expresses  a  large  quantity,  amount,  extent,  or  degree. 
“Much”  quantifies  over  mass  nouns,  as  opposed  to  count  nouns. 

Eg,  List  the  departments  vhich  have  [much  of  the  computer  expertise] . 
Eg,  Terrorists  burned  [much  information  about  their  activities]  in 
an  attack  on  the  Government  House. 

Eg,  [Much  of  the  dynamite  at  the  presidential  farm]  vas  stolen  by 
guerrillas. 

Criteria:  Demonstrated  understanding  that  the  existential  quantifier  “much”  expresses  a 
large  amount  of  the  substance  expressed  by  the  mass  noun  which  it  quantifies. 

_ Score: _ 


2. 2. 3. 3  Comparative  (Covered  in  Section  VII,  Comparatives) 

The  existential  quantifier  “more”  is  the  comparative  form  (denoting  a  higher  degree  of  quan¬ 
tity)  of  both  “many”  and  “much”.  “More”  expresses  a  greater  number  or  amount;  an  addi¬ 
tional  amount. 

2. 2. 3.4  Superlative 

The  existential  quantifier  “most”  is  the  superlative  form  (denoting  the  highest  degree  of 
quantity)  of  both  “many”  and  “much”.  “Most”  expresses  greatest  number  or  amount. 

Eg,  List  the  department  vith  [the  most  doctorates] . 

Eg,  What  are  [most  of  the  vomen  paid]? 

Eg,  Tventy  Cabinet  Members  vere  killed  yesterday.  [Most  Cabinet 
Members]  vere  killed  by  a  carbomb. 
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Eg,  Terrorists  killed  [most  of  the  peasants  in  Cuilapa] . 

Eg,  [Most  of  the  dynamite  sticks  at  the  presidential  farm]  were 
stolen  by  guerrillas. 

Criteria:  Demonstrated  understanding  that  the  existential  quantifier  “most”  expresses  high 
est  number  or  greatest  amount. 

_ Score: _ 


2.2.4  Paucal 

“ PaucaT  means  a  small  amount. 

2.2.4. 1  FEW 

The  existential  quantifier  “few”  expresses  at  least  some  but  a  small  amount  in  number. 
“ Few ”,  as  opposed  to  “a  few”,  signals  a  sharp  contrast  between  the  group  picked  out  by  the 
quantifier  and  some  other  group  not  picked. 

Since  “a  few”  determines  plural  count  nouns,  which  never  occur  alone  with  the  indefinite 
article  (  “a”),  we  treat  the  article  as  belonging  to  the  quantifier. 

Test  the  quantifier  “few”,  either  with  or  without  the  article  “a”. 

Eg,  What  is  the  department  with  [few  of  the  women]? 

Eg,  List  [a  few  employees  in  department  7] . 

Eg,  What  high  salary  do  [few  employees]  earn? 

Eg,  What  product  do  [a  few  salespersons]  sell? 

Eg,  [A  few  of  the  terrorists  in  ARENA]  attacked  the  Government 
House . 

Eg,  Twenty  Cabinet  Members  were  killed  yesterday.  [Few  of  the 
Cabinet  Members]  were  killed  by  a  carbomb. 

Eg,  Terrorists  killed  [few  peasants]  in  Cuilapa. 

Eg,  ARENA  kidnapped  [a  few  Government  Officials] . 

Eg,  [A  few  peasants]  were  injured  during  an  attack  by  guerrillas. 
Eg,  [Few  of  the  dynamite  sticks  at  the  presidential  farm]  were 
stolen  by  guerrillas. 


Criteria:  Demonstrated  understanding  that  the  existential  quantifier  “a  few”  expresses  at 
least  some  but  a  small  amount  in  number. 


Score: 


2.2.4.2  LITTLE 
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The  existential  quantifier  “ little ”  expresses  small  in  quantity  or  degree.  “ Little ”  quantifies 
over  mass  nouns. 

Eg,  What  is  the  department  vith  [little  of  the  management  expertise]? 
Eg,  Terrorists  burned  [little  information]  in  the  attack  on  the 
Government  House. 

Eg,  [Little  of  the  dynamite  at  the  presidential  farm]  was  stolen 
by  guerrillas. 

Criteria:  Demonstrated  understanding  that  the  existential  quantifier  u little  ”  expresses  small 
in  quantity  or  degree. 

_ Score: _ 


2.2.4. 3  Comparative  (Covered  in  Section  VII,  Comparatives) 

The  existential  quantifiers  “fewer”  and  “less”  are  the  comparative  forms  of  “few”  and  “little”, 
respectively.  They  both  express  a  smaller  amount. 

2.2.4.4  Superlative 

The  existential  quantifiers  “fewest”  and  “least ”  are  the  superlative  forms  of  “few”  and  “little”. 
They  express  the  smallest  number  or  amount.  Fewest  is  used  to  quantify  count  nouns  while 
least  is  used  to  quantify  mass  nouns. 

Test  one  of  the  paucal  superlative  quantifiers  fewest  or  least. 

Eg.  List  the  department  with  [the  fewest  employees] . 

Eg,  What  department  has  [fewest  of  the  PhDs]? 

Eg,  What  is  the  department  with  [the  least  of  the  management 
expertise]? 

Eg,  ARENA  killed  [fewest  of  the  Government  Officials] . 

Criteria:  Demonstrated  understanding  that  the  existential  quantifiers  “fewest”  and  “least” 
express  the  smallest  number  or  amount. 

_ Score: _ 


2.3  Existential  Non- Assertive  Quantifiers 
2.3.1  ANY 

The  existential  quantifier  “any”  expresses  one  or  another  taken  at  random  from  a  totality; 
every. 

There  are  two  senses  to  the  word  “any”.  An  exclusive  sense,  meaning  one  out  of  the  total, 
is  exemplified  in  the  sentence: 
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Pick  any  number  from  one  to  ten. 

An  inclusive  sense  of  “ any ”,  meaning  all,  is  exemplified  in  the  following  sentence: 

Any  employee  who  has  worked  for  25  years  is  a  senior  employee. 

Test  the  existential  quantifier  “any”  with  either  the  inclusive  or  exclusive  sense. 


Eg,  List  [any  information  about  John  Smith] . 

Eg,  Do  [any  employees]  earn  $20,000  ? 

Eg,  List  [any  of  the  employees  in  the  New  York  branch] . 

Eg,  Terrorists  did  not  kill  [any  of  the  officials]  in  the  bombing  of 
the  Government  House. 

Eg,  [Any  official  at  the  Government  House]  vas  killed  in  the  bombing. 

Eg,  Terrorists  killed  [any  government  official  that  they  saw]. 

Eg,  The  URNG  guerrillas  burned  [any  farm  along  the  road  to  Guatemala 
City]  on  February  5. 

Eg,  The  URNG  stole  [any  dynamite  they  found  at  the  presidential  farm] 
in  a  February  2  attack. 

Eg,  The  URNG  did  not  kidnap  [any  of  the  peasants  at  the  Santo 
Tomas  farm] . 


Criteria:  Demonstrated  understanding  that  the  existential  quantifier  “any”  expresses  one  or 
another  object  taken  at  random  from  a  totality  or  every  one  of  the  objects. 

_ _ Score: _ 


2.3.2  EITHER 

The  existential  quantifier  “either”  expresses  one  or  the  other  of  two. 

There  are  two  senses  to  the  word  “either”.  An  exclusive  sense  of  either,  meaning  one  or  the 
other,  but  not  both ,  is  exemplified  in  the  sentence: 

Take  either  road. 

An  inclusive  sense,  meaning  one  and  the  other  of  two ,  is  exemplified  in  the  sentence: 

There  were  flowers  blooming  on  either  side  of  the  walk. 

We  will  not  differentiate  between  the  inclusive  and  exclusive  senses  when  testing  either.  In 
order  to  test  that  either  expresses  two,  we  will  test  with  a  successful  presupposition  and  with 
a  failed  presupposition. 
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2.3.2. 1  Successful  Presupposition 

Successful  presupposition  occurs  in  sentences  where  there  is  a  presupposition  which  is  not 
mistaken.  For  example,  in  the  sentence 

List  either  research  assistant  in  department  1 7. 

it  is  presupposed  that  there  are  exactly  two  research  assistants.  If  the  presupposition  holds, 
it  is  called  a  “ successful  presupposition”. 

Test  “ either ”  as  a  quantifier  for  a  noun  phrase  which  expresses  exactly  two  objects. 


Eg,  What  is  [either  of  the  women  in  New  York]  paid? 

Eg,  What  commission  did  [either  Smith]  earn  in  1989? 

Eg,  Terrorists  did  not  kill  [either  of  the  Cabinet  Members]  in  the 
bombing  of  the  Government  House. 

Eg,  [Either  Marxist  group]  bombed  the  Government  House. 

Eg,  [Either  of  the  URNG  units]  will  claim  responsibility  for  an 
attack  on  the  presidential  farm. 


Criteria:  Demonstrated  understanding  that  “either”  expresses  one  or  the  other  of  two  objects, 
exclusively  or  inclusively. 

_ _ Score: - 


2. 3. 2. 2  Failed  Presupposition 

Test  whether  the  system  recognizes  the  error  when  “either”  quantifies  a  noun  phrase  which 
refers  to  a  set  of  objects  whose  cardinality  is  not  two. 


Eg,  What  is  [either  of  the  Chicago  women]  paid? 

Eg,  What  commission  did  [either  Jones]  earn  in  1989? 

Eg,  Terrorists  did  not  kill  [either  of  the  Prime  Minister]  in  the 
bombing  of  the  Government  House. 

Eg,  Arena  kidnapped  3  men,  but  did  not  kill  [either  of  the  men] . 


Criteria:  Demonstrated  understanding  of  the  discrepancy  between  the  existential  quantifier 
“either”  and  the  cardinality  of  the  set  of  objects  expressed  by  the  noun  phrase  which  it 
quantifies  (in  this  case  not  exactly  two). 

_ . _ Score: _ 


2.4  Negative  Quantifiers 
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According  to  the  MUC  rules,  templates  are  only  generated  for  reports  of  terrorist  activities. 
Since  a  sentence  with  a  negative  quantifier  is  often  a  report  of  the  non-occurrence  of  such  an 
incident,  a  MUC  system  might  not  create  a  template  for  such  an  input.  In  the  case  of  such  an 
event,  it  is  unclear  whether  the  system  has  successfully  understood  negative  quantification. 

2.4.1  NO 

The  negative  quantifier  “no”  expresses  not  any. 

Test  the  negative  quantifier  “no”. 

Eg,  List  the  employees  with  [no  degree] . 

Eg,  What  branch  includes  [no  women]? 

Eg,  Terrorists  killed  [no  person]  in  the  attack  on  the  Government 
House. 

Eg,  [No  people]  were  killed  in  the  attack  on  the  Government  House. 
Eg,  The  URNG  attacked  [no  presidential  farm]  on  February  5. 


Criteria:  Demonstrated  understanding  that  the  negative  quantifier  “no”  expresses  not  any 
or  a  zero  amount. 

_ Score: - 


2.4.2  NONE 

The  negative  quantifier  “none”  expresses  not  any. 
Test  the  negative  quantifier  “none”. 


Eg,  What  department  includes  [none  of  the  women]? 

Eg,  Terrorists  killed  [none  of  the  officials]  in  the  bombing  of 
the  Government  House. 

Eg,  [None  of  the  Cabinet  Members]  were  killed  by  the  carbomb. 

Eg,  [None  of  the  peasants  at  the  Santo  Tomas  farm]  were  kidnapped 
by  the  guerrillas. 


Criteria:  Demonstrated  understanding  that  the  negative  quantifier  “none "expresses  not  any 
or  a  zero  amount. 


2.4.3  NEITHER 


Score: 


The  negative  quantifier  “neither”  expresses  not  the  one  or  the  other  of  two  or  more,  chiefly 
limited  to  two. 


Test  the  negative  quantifier  “neither”. 
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Eg,  What  Nev  York  salary  does  [neither  of  the  two  women  in  New  York] 
earn? 

Eg,  What  products  does  [neither  Smith]  sell? 

Eg.  List  the  branch  with  [neither  of  the  Smiths] . 

Eg,  [Neither  of  the  Cabinet  Members]  was  killed  by  the  carbomb. 

Eg,  Terrorists  killed  [neither  Senator]  in  the  attack  on  the 
Government  House. 

Eg,  The  URNG  attacked  [neither  president ial  farm]  on  February  5. 

Criteria:  Demonstrated  understanding  that  the  negative  quantifier  “ neither ”  expresses  a 
zero  amount  of  two  or  more. 

- _ Score: _ 


2.5  Numerical  Quantifiers 
2.5.1  Cardinal  Numbers 

Cardinal  numbers  (  “one”,  “two”,  “three”,  . . . )  may  act  as  quantifiers  and  either  follow  the 
definite  article  li.he”  or  do  not  follow  an  article  at  all.  When  they  are  used  with  the  definite 
article,  as  in: 

List  the  one  research  assistant  in  department  17. 

it  is  presupposed  that  there  are  exactly  that  number  of  objects.  A  failed  presupposition  occurs 
when  there  are  less  than  or  more  than  the  number  specified. 

When  testing  cardinal  numbers  with  an  article,  make  sure  that  the  presupposition  holds. 

Eg,  List  [one  women  in  the  New  York  branch] . 

Eg,  What  does  [one  of  the  women  in  Chicago]  earn? 

Eg,  What  do  [the  three  Chicago  employees]  earn? 

Eg,  What  amount  do  [the  two  Smiths]  earn  in  commission? 

Eg,  [One  terrorist]  attacked  the  Government  House. 

Eg,  [Five  of  the  Cabinet  Members]  were  killed  by  a  carbomb. 

Eg,  Terrorists  killed  [six  Government  Officials]  in  the  bombing  of 
the  Government  House. 

Eg,  Terrorists  killed  [the  five  Cabinet  Members]  in  the  bombing  of 
the  Government  House. 

Eg,  [The  two  presidential  farms]  were  attacked  by  the  URNG. 

Criteria:  Demonstrated  understanding  of  the  quantification  imposed  on  the  noun  phrase  by 
the  cardinal  number. 


Score: 
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2.5.2  Ordinal  Numbers 


Ordinal  numbers  express  a  position  within  an  ordering  of  the  elements  of  a  group  of  objects. 
The  ordinal  numbers  are  “first”,  “second”,  “third”,... 

In  the  sentence 

Smith  was  the  third  employee  to  be  hired. 

the  ordering  is  with  respect  to  hiring  times.  It  is  presupposed  that  at  least  three  employees 
have  been  hired. 

In  contrast  to  a  cardinal  number  which  specifies  the  number  of  elements  in  a  set,  an  ordinal 
number  specifies  one  individual  from  an  ordered  set.  For  example,  “four  people”  refers  to  a 
set  containing  four  elements  while  “fourth  person”  refers  to  the  fourth  in  a  sequence  of  four 
or  more. 


Eg,  List  the  [first  employee  on  the  payroll] . 

Eg,  Who  was  the  [third  employee  to  be  hired]  ? 

Eg,  The  [second  attack  on  the  Government  House]  killed  three  people. 
Eg,  ARENA  killed  the  [first  man  they  saw] . 

Eg,  ARENA  kidnapped  two  men  and  killed  the  [second  of  them] . 

Eg,  [The  first  of  three  guerrilla  attacks]  occurred  on  February  2. 

Criteria:  Demonstrated  understanding  of  the  ordering  of  the  group  of  objects  expressed  by 
the  noun  phrase  that  the  quantifier  modifies  and  chose  the  correct  individual  from  that  group 
as  specified  by  the  ordinal  number. 

_ Score:  _ 


2  5.3  Fractions 

The  fractions  include  “(one-)half” ,  “one-third” ,  “one-fourth”,  “one  quarter”,  “one-fifth”, 
and  so  on.  Fractions  express  a  portion  of  the  entity  expressed  by  the  quantified  noun  phrase. 

Test  a  fraction. 

Eg,  What  is  [half  the  salary  of  John  Smith]? 

Eg,  What  is  [one-fourth  the  number  of  employees]  ? 

Eg,  Terrorists  killed  [half  the  border  patrol  at  the  Nicaraguan 
border] . 

Eg,  (Preliminary  Sentence:  Terrorists  attacked  twelve  border  guards 
at  the  Nicaraguan  border.)  The  terrorists  killed  [half  of  the 
guards] . 

Eg,  (Preliminary  Sentences:  Ten  farms  were  attacked  by  guerrillas.) 
[Half  of  the  attacked  farms]  were  burned  to  the  ground. 
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Criteria:  Demonstrated  understanding  of  the  quantification  imposed  on  the  noun  phrase  by 
the  fraction  quantifier. 


_ _ Score: _ 

2.5.4  Multipliers 

The  multipliers  include  “ twice ”,  “ thrice ”,  “ two  times”,  “three  times”,  “ four  times”,  and 
so  on.  Multipliers  express  a  multiplication  of  the  entity  expressed  by  the  quantified  noun 
phrase. 

Test  a  multiplier. 


Eg,  List  [twice  the  salary  of  John  Smith]. 

Eg,  What  is  [three  times  the  salary  of  John  Smith]  ? 

Eg,  Two  Mayors  were  killed.  [Twice  that  number]  were  kidnapped. 

Eg,  (Preliminary  Sentence:  Five  farms  were  destroyed  by  guerrillas.) 
[Twice  that  number]  were  attacked  by  the  URNG. 


Criteria:  Demonstrated  understanding  of  the  quantification  imposed  on  the  noun  phrase  by 
the  multiplier. 

_ _ Score:  _ 


3.  Indefinite  Quantifier  Pronouns 

The  indefinite  quantifier  pronouns  (  “everybody”,  “everyone”,  “ everything ”,  “somebody”, 
“someone”,  “something”,  “anybody”,  “anyone”,  “anything”,  “nobody”,  “no  one”,  “nothing”) 
serve  as  the  head  of  a  NP. 

We  have  already  seen  quantifiers  (  “all”,  “some”,  “any”,  “none”,  and  so  on)  which  can  be 
used  pronominally,  as  the  head  of  a  noun  phrase.  These  are  often  followed  by  a  prepositional 
phrase  beginning  with  the  preposition  “ of ”,  called  the  “partitive  of”.  Indefinite  quantifier 
pronouns  are  never  used  with  a  partitive  of.  They  can  only  be  used  as  the  head  of  a  noun 
phrase  and  are  never  used  in  a  determinative  position. 

In  this  subsection  we  will  test  each  indefinite  quantifier  pronoun  individually.  Below  are  some 
suggested  sentence  patterns.  In  each  test,  you  will  replace  the  [Indef.  Quant.  Pronoun]  with 
the  specified  determiner  quantifier. 

MAIN  QUANTIFIER  PRONOUN  PATTERNS: 

Pattern:  List  [Indef  Quant  Pronoun]  [Postmodification]  . 

Example:  List  everyone  who  earns  above  $45,000. 

Pattern:  List  [NP]  with  [Indef  Quant  Pronoun]  [Postmodification]. 

Example:  List  the  departments  with  anyone  who  earns  below  $25,000 


Pattern:  Who  [BE- Verb]  [Indef  Quant  Pronoun]  [Postmodification]? 

Example:  Who  is  everybody  in  department  12? 

Pattern:  What  ([NP])  [BE- Verb]  [Indef  Quant  Pronoun]  [Postmodification]  [Verb  (-ed)]? 

Example:  What  salary  is  everybody  in  department  12  paid? 

Pattern:  What  ([NP])  [DO- Verb]  [Indef  Quant  Pronoun]  [Postmodification]  [Verb]? 

Example:  What  product  does  everyone  in  department  17  sell? 


Pattern:  [NP]  [Verb]  [Indef  Quant  Pronoun]  [Postmodification]. 

Example:  Terrorists  killed  anyone  at  the  Government  House. 

Pattern:  [Indef  Quant  Pronoun]  [Postmodification]  [VP]  . 

Example:  Nobody  in  the  FMLN  killed  a  government  official. 


In  the  examples  in  the  following  subsections  the  indefinite  quantifier  pronoun  is  enclosed  in 
brackets  for  ease  of  identification. 


3.1  Universal 

3.1.1  Personal 

The  universal  personal  indefinite  quantifier  pronouns  are  “everyone”  and  “everybody”,  mean¬ 
ing  every  person. 

Replace  [Indef  Quant  Pronoun]  in  the  MAIN  QUANTIFIER  i  ROTO  JN  PATTERNS  with: 
everyone  or  everybody 

Eg.  List  [everybody]  who  earns  above  $45,000. 

Eg,  Who  is  [everyone]  in  the  New  York  branch? 

Eg,  What  salary  is  [everybody]  in  Chicago  paid? 

Eg,  What  manager  supervises  [everyone]? 

Eg,  What  salaries  does  [everybody]  in  department  17  earn? 

Eg,  Terrorists  killed  [everyone]  in  the  Government  House. 

Eg,  [Everybody]  at  the  Government  House  was  killed  in  a  terrorist 
attack. 

Eg,  [Everyone]  was  killed  in  the  URNG  attack  on  the  Santo  Tomas 
farm. 

Criteria:  Demonstrated  understanding  that  the  universal  personal  indefinite  quantifier  pro¬ 
noun  “everyone”  or  “everybody”  means  every  person. 

_ Score:  _ 


3.1.2  Nonpersonal 

The  universal  nonpersonal  indefinite  quantifier  pronoun  is  “ everything ”,  meaning  all  that 
exists. 

Replace  [Indef  Quant  Pronoun]  in  the  MAIN  QUANTIFIER  PRONOUN  PATTERNS  with: 
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everything 


Eg,  List  [everything]  that  employees  sell. 

Eg,  What  is  [everything]  that  department  12  produces? 

Eg,  Terrorists  bombed  [everything]  at  the  Government  House. 

Eg,  [Everything]  at  the  Government  House  was  destroyed  in  a  terrorist 
attack. 

Eg,  [Everything]  was  destroyed  in  the  URNG  attack  on  the  Santo  Tomas 
farm. 

Criteria:  Demonstrated  understanding  that  the  universal  nonpersonal  indefinite  quantifier 
pronoun  “ everything ”  means  all  that  exists. 

_ _ Score: _ 


3.2  Existential  Assertive 

3.2.1  Personal 

The  existential  assertive  personal  indefinite  quantifier  pronouns  are  “someone”  and  “some¬ 
body”,  meaning  one  or  some  person  of  unspecified  or  indefinite  identity. 

Replace  [Indef  Quant  Pronoun]  in  the  MAIN  QUANTIFIER  PRONOUN  PATTERNS  with: 

someone  or  somebody 

Eg,  List  [somebody]  who  earns  above  $45,000. 

Eg,  What  salary  is  [someone]  in  the  New  York  branch  paid? 

Eg,  What  items  does  [somebody]  in  the  New  York  branch  sell? 

Eg,  Terrorists  killed  [someone]  in  the  Government  House. 

Eg,  [Somebody]  at  the  Government  House  was  killed  in  a  terrorist 
attack. 

Eg,  [Someone]  was  killed  in  the  URNG  attack  on  the  Santo  Tomas 
farm. 

Criteria:  Demonstrated  understanding  that  the  existential  assertive  personal  indefinite  quan¬ 
tifier  pronoun  “someone”  or  “somebody”  means  one  or  some  person  of  unspecified  or  indefi¬ 
nite  identity. 

_ _ _ Score: _ 

3.2.2  Nonpersonal 

The  existential  assertive  nonpersonal  indefinite  quantifier  pronoun  is  “ something ”,  meaning 
some  indeterminate  or  unspecified  thing. 

Replace  [Indef  Quant  Pronoun]  in  the  MAIN  QUANTIFIER  PRONOUN  PATTERNS  with: 
something 
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Eg.  List  [something]  that  the  Nev  York  branch  sells. 

Eg,  What  is  [something]  that  the  Nev  York  branch  sells? 

Eg,  What  price  does  [something]  cost? 

Eg,  Terrorists  bombed  [something]  at  the  Government  House. 

Eg,  [Something]  at  the  Government  House  was  destroyed  in  a  terrorist 
attack . 

Eg,  [Something]  vas  destroyed  in  the  URNG  attack  on  the  Santo  Tomas 
farm. 

Criteria:  Demonstrated  understanding  that  the  existential  assertive  nonpersonal  indefinite 
quantifier  pronoun  “something”  means  some  indeterminate  or  unspecified  thing. 

_ _ Score: _ 


3.3  Existential  Nonassertive 

3.3.1  Personal 

The  existential  nonassertive  personal  indefinite  quantifier  pronouns  are  “anyone”  and  “any¬ 
body”,  meaning  any  person. 

Replace  [Indef  Quant  Pronoun]  in  the  MAIN  QUANTIFIER  PRONOUN  PATTERNS  with: 
anyone  or  anybody 

Eg.  List  [anyone]  vho  is  paid  $35,000. 

Eg,  What  items  does  [anybody]  in  the  New  York  branch  sell? 

Eg,  What  benefits  does  [anyone]  vho  is  salaried  receive? 

Eg,  Terrorists  killed  [anybody]  at  the  Government  House. 

Eg,  [Anyone]  vho  was  at  the  Government  House  was  killed  by 
terrorists. 

Criteria:  Demonstrated  understanding  that  the  existential  nonassertive  personal  indefinite 
quantifier  pronoun  “anyone”  or  “anybody”  means  any  person. 

_ Score: _ 

3.3.2  Nonpersonal 

The  existential  nonassertive  nonpersonal  indefinite  quantifier  pronoun  is  “anything”,  mean¬ 
ing  any  thing  whatever. 

Replace  (Indef  Quant  Pronoun]  in  the  MAIN  QUANTIFIER  PRONOUN  PATTERNS  with: 
anything 

Eg.  List  [anything]  that  is  produced  by  Department  17. 

Eg,  [Anything]  that  vas  at  the  Government  House  vas  destroyed  by 
terrorists. 
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Criteria:  Demonstrated  understanding  that  the  existential  nonassertive  nonpersonal  indefi¬ 
nite  quantifier  pronoun  “ anything ”  means  any  thing  whatever. 

_  Score:  _ 


3.4  Negative 

According  to  the  MUC  rules,  templates  are  only  generated  for  reports  of  terrorist  activities. 
Since  a  sentence  with  a  negative  indefinite  quantifier  pronoun  is  often  a  report  of  the  non¬ 
occurrence  of  such  an  incident,  a  MUC  system  might  not  create  a  template  for  such  an  input. 
In  the  case  of  such  an  event,  it  is  unclear  whether  the  system  has  successfully  understood 
the  negative  indefinite  quantifier  pronoun. 

3.4.1  Personal 

The  negative  personal  indefinite  quantifier  pronouns  are  “no  one ”  and  “ nobody ”,  meaning 
no  person;  not  anybody. 

Replace  [Indef  Quant  Pronoun]  in  the  MAIN  QUANTIFIER  PRONOUN  PATTERNS  with: 
no  one  or  nobody 

Eg,  What  salary  does  [nobody]  in  the  New  York  branch  earn? 

Eg,  What  product  did  [no  one]  in  Chicago  sell  in  1990? 

Eg,  [No  one]  was  killed  in  the  bombing  at  the  Government  House. 

Eg,  [Nobody]  in  the  FMLN  killed  a  government  official. 

Eg.  [Nobody]  was  injured  in  the  URNG  attack  on  the  Santo  Tomas 
farm . 


Criteria:  Demonstrated  understanding  that  the  negative  personal  indefinite  quantifier  pro¬ 
noun  “no  one”  or  “nobody”  means  no  person;  not  anybody. 

_ _ _ Score: _ 

3.4.2  Nonpersonal 

The  negative  nonpersonal  indefinite  quantifier  pronoun  is  “nothing”,  meaning  not  any  thing; 
no  thing. 

Replace  [Indef  Quant  Pronoun]  in  the  MAIN  QUANTIFIER  PRONOUN  PATTERNS  with: 
nothing 


Eg,  What  employees  earn  [nothing]  from  commissions? 

Eg,  [Nothing]  was  destroyed  in  the  bombing  at  the  Government  House. 
Eg,  Terrorists  destroyed  [nothing]  in  the  bombing  at  the  Government 
House. 

Eg,  [Nothing]  was  damaged  in  the  URNG  attack  on  the  Santo  Tomas 
farm. 
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Criteria:  Demonstrated  understanding  that  the  negative  nonpersonal  indefinite  quantifier 
pronoun  “nothing ”  means  not  any  thing ;  no  thing. 

_ Score: _ 


4.  Existential  There 

Existential  there  sentences  express  the  existence  of  some  object  specified  by  the  noun  phrase 
which  serves  as  the  subject  complement  of  the  sentence.  The  word  “there”  in  existential 
there  sentences  is  not  used  in  a  locative  sense. 

The  basic  structure  of  an  existential  there  sentence  is  shown  below,  with  [BE- Verb]  being  a 
form  of  the  copular  verb  “to  be”,  possibly  with  an  auxiliary  verb  or  verbs. 

Pattern:  There  [BE-Verb]  [NP]. 

Example:  There  is  a  director. 

There  are  five  new  employees  in  Department  77. 

In  this  section,  several  sentence  types  are  described  which  employ  an  existential  there.  Make 
sure  that  a  singular  existential  sentence  is  tested  as  well  as  a  plural  existential  sentence. 

Note  that  a  noun  phrase  includes  optional  postmodification. 

4.1  Declaratives 

Pattern:  There  [BE-Verb]  [NP]. 

Examples:  There  are  salespersons  in  the  New  York  office. 

There  are  electric  facilities  that  the  terrorists  attacked. 

There  have  been  two  URNG  attacks  on  presidential  farms  outside  of 
Guatemala  City  today. 

Criteria:  Demonstrated  understanding  that  the  existential  there  declarative  sentence  spec¬ 
ifies  the  existence  of  the  object  expressed  by  the  noun  phrase  which  serves  as  the  subject 
complement  of  the  sentence. 

_ _  Score: _ 


4.2  In  Relative  Clauses 

Pattern:  [NP]  [VP]  (that)  there  [BE-Verb]. 

Example:  The  terrorists  attacked  the  electric  that  there  are. 

facilities 

Pattern:  List  [NP]  (that)  there  [BE-Verb]. 

Examples:  List  the  financial  goal  for  1991  that  there  is. 

List  the  departments  that  there  are. 
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Pattern:  [NP]  (that)  there  [BE- Verb]  [VP]  . 

Example:  All  the  presidential  farms  that  there  are  were  attacked  by 

on  the  road  to  Guatemala  City  URNG  guerrillas. 

Criteria:  Demonstrated  understanding  that  the  existential  there  relative  clause  specifies  the 
existence  of  the  object  expressed  by  the  noun  phrase  which  it  postmodifies. 

_ Score: _ 


4.3  What  Questions 


Pattern: 

What 

[NP]  ([Postmodification]) 

[BE-Verb] 

there  ? 

Examples: 

What 

job  with  the  least  training 

is 

there? 

What 

positions  in  department  17 

are 

there? 

Pattern: 

What 

[NP] 

[BE- Verb]  there 

([Postmodification])  ? 

Examples: 

What 

employees 

are 

there 

in  department  17? 

What 

presidential  farms 

are 

there 

on  the  road  to  Guatemala  City? 

Criteria:  Demonstrated  understanding  that  the  existential  there  clause  in  the  what  question 
refers  to  the  existence  of  the  object  expressed  by  the  noun  phrase  which  it  follows. 


Score: 


4.4  Yes-No  Questions 


Pattern: 

Examples: 


[BE-Verb] 

Is 

Are 

Are 


there  [NP]  ? 

there  a  Boston  branch  ? 

there  any  sales  departments? 

there  any  guerrilla  groups  that  have  claimed  responsibility 
for  the  attacks  earlier  today  on  presidential  farms? 


Criteria:  Demonstrated  understanding  that  the  existential  there  clause  in  the  yes-no  question 
refers  to  the  existence  of  the  object  expressed  by  the  noun  phrase  which  follows  it. 


Score: 


4.5  Tag  Questions 

A  Tag  Question  is  a  type  of  yes-no  question  conveying  positive  or  negative  orientation  relative 
to  the  statement  to  which  it  is  appended. 

The  tag  is  formed  in  three  steps.  First,  the  statement’s  auxiliary  verb,  if  there  is  one,  is 
repeated.  If  there  is  no  auxiliary,  the  dummy  auxiliary  DO  is  used.  Second,  if  the  question 
or  statement  is  positive,  a  negation  is  added.  If  the  question  or  statement  is  negative,  no 
negation  is  use  in  the  tag.  Third,  the  statement’s  subject  is  repeated.  To  form  a  tag  for  the 
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sentence 


The  employees  are  paid  well. 

we  take  the  auxiliary  verb  ( “are”),  append  the  negation  (  “aren’t”),  and  repeat  the  subject 
(  “aren’t  they”).  The  tag  is  then  appended  to  the  original  sentence,  resulting  in  the  following: 

The  employees  are  paid  well,  aren’t  they? 

In  the  case  of  a  tag  question  using  am  existential  there  construction,  the  auxiliary  verb, 
usually  a  form  of  the  [BE- Verb],  is  negated  and  the  word  “there”  is  repeated  as  the  subject. 
To  form  a  tag  for  the  sentence 

There  are  five  employees. 

we  take  the  auxiliary  verb  ( “are”),  append  the  negation  ( “aren’t”),  and  repeat  the  subject 
(  “aren’t  there”).  The  tag  is  then  appended  to  the  original  sentence,  resulting  in  the  following: 

There  are  five  employees,  aren’t  there? 


Test  an  existential  there  sentence. 


Pattern:  There  [BE-Verb]  [NP], 

Examples:  There  is  a  sales  department, 

There  are  female  salespersons, 

There  have  been  guerrilla  attacks  on  the 

presidential  farms  outside 
of  Guatemala  City  today, 


[Tag  Question]  ? 
isn’t  there? 
aren’t  there? 
haven’t  there? 


Criteria:  Demonstrated  understanding  that  the  tag  questions  the  existence  of  the  object 
expressed  by  the  noun  phrase  which  serves  as  the  subject  complement  of  the  declarative 
existential  there  sentence. 


Score: 


5.  Universal  Adjectives 

Universal  Adjectives  are  a  set  of  open  class  quantifiers  such  as  whole,  entire,  or  full. 
They  occur  in  the  position  of  premodification  of  a  noun  phrase.  In  the  following  example 
sentences,  the  noun  phrase  which  is  premodified  with  a  universal  adjective  will  be  bracketed 
for  greater  readability. 

Eg,  List  [the  entire  staff  of  department  17] . 

Eg.  Terrorists  bombed  [the  entire  Government  House] . 

Eg,  [The  complete  electric  facility]  was  destroyed  by  terrorists. 

Eg,  [The  entire  store  of  dynamite]  was  stolen  by  the  URNG  guerrillas. 
Eg,  [A  great  deal  of  dynamite]  was  stolen  by  the  URNG  guerrillas. 
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Criteria:  Demonstrated  understanding  that  the  universal  adjective  indicates  the  quantity  or 
amount  for  the  noun  phrase  which  it  modifies. 

_ _ Score: _ 

6.  Open-Class  Determiner  Quantifiers  (t.b.d) 

Open-class  determiner  quantifiers  are  phrases  which  act  like  determiners  in  that  they  must 
precede  nouns,  and  they  can  also  precede  other  determiners.  This  group  includes  phrases 
such  as: 

A  great  deal  of 
lots  of 

a  small  amount  of 
a  great  number  of 
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VII.  COMPARATIVES 


A  comparative  expresses  a  comparison  or  contrast  between  two  events,  entities  or  attributes 
with  respect  to  some  measure  on  a  scale,  which  may  be  numerical  (age,  height)  or  non- 
numerical  (friendliness,  perseverance).  In  the  sentence: 

Jill  has  more  experience  than  Ted. 

the  comparison  is  between  Bill  and  Ted  with  respect  to  years  of  experience ,  a  numerical 
quantity.  In  the  sentence: 

Tom.  is  less  friendly  than  his  brother. 

the  comparison  is  between  Tom  and  his  brother  with  respect  to  friendliness,  which  is  not 
measured  numerically. 

A  comparative  sentence  has  two  clauses  connected  by  a  comparative  conjunct  such  as  the 
word  “ than ”,  or  the  phrase  “as  . ..  as”.  The  two  clauses  have  roughly  the  same  structure, 
with  the  corresponding  elements  in  the  clauses,  labeled  [<element>]i  and  [ <element>]2 , 
either  matching  or  differing.  There  must  be  at  least  one  pair  which  differs,  however,  in  order 
to  make  a  comparison.  In  this  next  sentence: 

Pattern:  What  [NP]j  [BE- Verb]  [Adj  +compare]i  than  [NP]2  [BE- Verb]  [Adj]2? 

Example:  What  employee  is  older  than  Smith  is  old? 

the  differing  items  are  John  Smith  and  some  yet  unnamed  employee .  The  other  elements  do 
not  differ  and  so  are  not  in  contrast. 

We  may  also  have  gapping  or  ellipsis.  This  means  a  grammatical  element  that  occurs  in  the 
first  clause  is  omitted  from  the  second.  The  element  in  the  first  clause  which  no  longer  has  a 
corresponding  element  in  the  second  clause  is  understood  to  be  present  in  the  second  clause 
and  is  not  the  item  being  contrasted. 

For  example,  we  may  omit  the  [BE- Verb]  and  the  adjective  in  the  sentence  above,  so  that  it 
reads: 

Pattern:  What  [NP^  [BE- Verb]  [Adj  +compare]i  than  [NP]2? 

Example:  Wha*  employee  is  older  than  Smith? 

Here,  the  sentence  is  understood  to  be  an  abbreviated  version  of  “ What  employee  is  older 
than  John  Smith  is  old ?”.  The  contrast  is  still  between  John  Smith  and  a  different  employee. 

In  the  following  sections  we  test  comparatives  with  no  gapping,  and  with  various  patterns 
of  gapping. 


1.  Comparative  Adjectives 

The  standard  comparative  adjective  pattern  is: 
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[NP]j  [BE- Verb]  [Adj  +compare]i  than  [NP]2  [BE- Verb]  [Adj]2 


1.1  Comparison  to  a  Higher  Degree 

A  comparative  expresses  a  comparison  or  contrast  between  two  entities  or  attributes  with 
respect  to  some  measure  on  a  scale,  which  may  be  numerical  or  non-numerical.  In  compara¬ 
tives  to  a  higher  degree,  the  measurement  of  the  first  entity  or  attribute  is  greater  than  that 
of  the  second.  Note  that  in  WH -questions,  a  WH- word  often  functions  as  a  noun  phrase  and 
expresses  an  unknown  entity.  In  comparing  two  entities,  one  of  the  entities  may  be  expressed 
by  a  W'TZ-word. 

A  comparative  to  a  higher  degree  is  formed  with  “[Adj  -(-compare]  than ”,  where  [Adj  -(-com¬ 
pare]  denotes  an  adjective  comparative  to  a  higher  degree.  An  adjective  comparative  to  a 
higher  degree  is  formed  as  follows:  (“ — *”  means  “becomes”) 

•  by  adding  an  -er,  with  associated  spelling  changes  if  needed: 

big  — ♦  bigger 

•  by  using  the  word  more ,  before  the  adjective: 

beautiful  — ►  more  beautiful 

“More”  can  be  used  as  a  quantifier  as  well  as  being  used  as  part  of  a  comparative  adjective 
phrase.  As  a  quantifier,  “more”  may  be  replaced  by  “a  greater  number  of”  or  “a  greater 
amount  of”.  Make  sure  that  “more”  is  not  used  as  a  quantifier  in  this  section  on  comparative 
adjectives. 

In  the  next  four  subsections  try  to  use  the  more  inflection  (marking)  at  least  once  and  the 
-er  inflection  at  least  once,  so  that  both  are  tested.  The  following  patterns  (which  may  be 
longer  than  one  line)  capture  the  structure  which  we  are  testing.  Use  these  patterns  in  the 
following  subsections  along  with  the  additional  instructions  in  each  subsection. 


MAIN  COMPARATIVE  ADJECTIVE  PATTERNS: 

Pattern  1:  List  [NP]x  [WH-Word]  [BE- Verb]  [Adj  +compare]i 

Example:  List  an  employee  who  is  more  experienced 


[NP]2  [BE-Verb] 

the  Supervisor  of  Department  77  is 


[Adj]2. 

experienced. 


than 

than 


Pattern  2: 
Example  a: 
Example  b: 


[WH-Word]  ([NP]0  [BE-Verb] 
What  employee  is 

Who  is 


[BE-Verb]  [Adj]2  ? 
is  experienced  ? 

is  old  ? 


[Adj  -(- compare]  1 

than 

[NP]2 

more  experienced 

than 

Jane  Doe 

older 

than 

John  Smith 
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Pattern  3:  [NPjj  [Rel.  Pronoun]  [BE- Verb]  [Adj  +compare]i  than 

Example:  The  terrorists  who  are  more  anticommunist  than 

[NP]a  [BE- Verb]  [Adj]2  [VP], 

ARENA  is  anticommunist  killed  the  Prime  Minister. 

Pattern  4:  [NP]j  [VP]  [Rel.  Pronoun]  [BE- Verb] 

Example:  The  terrorists  killed  the  Prime  Minister  who  is 

[Adj  +compare]i  than  [NP]2  [BE- Verb]  [Adj]2. 
more  pro-Cuban  than  he  is  pro-US. 

Pattern  5:  [NP]i  [BE- Verb]  [Adj  +compare]i  than  [NP]2 

Example:  (Preliminary  Sentence:  Guerrillas  attacked  the  presidential  farm.) 

The  attack  was  deadlier  than  the  previous  attack 

[BE- Verb]  [Adj]2. 
was  deadly. 

1.1.1  No  Gapping 

Test  a  comparative  adjective  to  a  higher  degree  with  no  gapping,  using  the  MAIN  COM¬ 
PARATIVE  ADJECTIVE  PATTERNS. 

Eg,  List  an  employee  who  is  more  experienced  than  the  Supervisor 
of  Department  77  is  experienced. 

Eg,  Who  is  older  than  John  Smith  is  old? 

Eg.  The  terrorists  who  are  more  anticommunist  them  ARENA  is  anti¬ 
communist  killed  the  Prime  Minister. 

Eg,  The  terrorists  killed  the  Prime  Minister  who  is  more  Marxist 
than  he  is  Maoist. 

Eg,  (Preliminary  Sentence:  Guerrillas  attacked  the  presidential  farm.) 
The  attack  was  deadlier  than  the  previous  attack  was  deadly. 

Criteria:  Demonstrated  understanding  of  either 

•  the  contrast  between  the  two  entities  expressed  by  [NP]i  and  [NP]2  with  respect  to 

the  attribute  or  attributes  expressed  by  [Adj  -(-compare]!  and  [Adj]2. 

or 

•  the  contrast  between  the  two  attributes,  expressed  by  [Adj  +compare]i  and  [Adj]2, 

of  one  entity  expressed  by  both  [NP]i  and  [NP]2. 

_  Score:  _ 


1.1.2  Adjective  Gapping 
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Test  a  comparative  adjective  to  a  higher  degree  with  adjective  gapping  by  omitting  the  [Adj]2 
from  the  MAIN  COMPARATIVE  ADJECTIVE  PATTERNS  as  shown  below: 

Pattern  1:  List  [NP]i  [WH-Word]  [B  E- Verb]  [Adj  -{-compare]  than 

Example:  List  an  employee  who  is  more  experienced  than 

[NP]2  [BE-Verb], 

the  Supervisor  of  Department  77  is. 

Pattern  2:  [WH-Word]  ([NP^)  [BE-Verb]  [Adj  +compare]  than  [NP]2 

Example  a:  What  employee  is  more  experienced  than  Jane  Doe 

Example  b:  Who  is  older  than  John  Smith 

[BE-Verb]? 

is? 


Pattern  3:  [NP]i  [Rel.  Pronoun]  [BE-Verb]  [Adj  -{-compare]  than 

Example:  The  terrorists  who  are  more  anticommunist  than 

[NP]2  [BE-Verb]  [VP]. 

ARENA  is  killed  the  Prime  Minister. 

Pattern  4:  [NP]i  [VP]  [Rel.  Pronoun]  [BE-Verb] 

Example:  The  terrorists  killed  the  Prime  Minister  who  is 

[Adj  {-compare]  than  [NP]2  [BE-Verb]. 
more  pro- Cuban  than  they  are. 

Pattern  5:  [NP]i  [BE-Verb]  [Adj  {-compare]]  than  [NP]2 

Example:  (Preliminary  Sentence:  Guerrillas  attacked  the  presidential  farm.) 

The  attack  was  deadlier  than  the  previous  attack 

[BE-Verb], 

was. 

Criteria:  Demonstrated  understanding  of  the  contrast  between  the  two  entities  expressed  by 
[NP]X  and  [NP]2  with  respect  to  the  attribute  expressed  by  [Adj  -(-compare]. 

_ Score: _ 


1.1.3  Copular  and  Adjective  Gapping 

Test  a  comparative  adjective  to  a  higher  degree  with  copular  and  adjective  gapping  by  omit¬ 
ting  the  [BE-Verb]  and  the  [Adj]2  from  the  MAIN  COMPARATIVE  ADJECTIVE  PAT¬ 
TERNS  as  shown  below: 
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Pattern  1: 
Example: 


Pattern  2: 
Example  a: 
Example  b: 


Pattern  3: 
Example: 


Pattern  4: 
Example: 


Pattern  5: 
Example: 


List  [NP]i  [WH-Word]  [BE- Verb]  [Adj  -I- com  pare]  than 

List  an  employee  who  is  more  experienced  than 

[NP]a. 

the  Supervisor  of  Department  77. 


[WH-Word] 

([NP],) 

[BE-Verb] 

[Adj  -(-compare] 

than 

[NP]2? 

What 

employee 

is 

more  experienced 

than 

Jane  Doe? 

Who 

is 

older 

than 

John  Smith? 

[NP]x  [Bel.  Pronoun]  [BE- Verb]  [Adj  -(-compare]  than 

The  terrorists  who  are  more  anticommunist  than 

[NP]2  [VP], 

ARENA  killed  the  Prime  Minister. 

[NP],  [VP]  [Rel.  Pronoun]  [BE- Verb] 

The  terrorists  killed  the  Prime  Minister  who  is 

[Adj  -(-compare]  than  [NP]2. 
more  pro-Cuban  than  they. 

[NP]]  [BE- Verb]  [Adj  -(-compare]]  than  [NP]2  . 

(Preliminary  Sentence:  Guerrillas  attacked  the  presidential  farm.) 

The  attack  was  deadlier  than  the  previous  attack. 


Criteria:  Demonstrated  understanding  of  the  contrast  between  the  two  entities  expressed  by 
[NP]j  and  [NP]2  with  respect  to  the  attribute  expressed  by  [Adj  -(-compare]. 

_  _ Score: _ 


1.1.4  Noun  Phrase  and  Copular  Gapping  (Pseudo-Comparatives) 

Test  a  comparative  adjective  to  a  higher  degree  with  noun  phrase  and  copular  gapping 
by  omitting  the  [NP]2  and  the  [BE- Verb]  from  the  MAIN  COMPARATIVE  ADJECTIVE 
PATTERNS  as  shown  below.  The  adjectives  must  be  different. 

Pattern  1:  List  [NP]i  [WH-Word]  [*.  -Verb]  [Adj  -(-compare]]  than 

Example:  List  an  employee  who  is  more  experienced  than 

[Adj]  2. 
educated. 
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Pattern  2:  [WH-Word]  ([NP],)  [BE- Verb]  [Adj  +compare],  than  [Adj]2  ? 

Example  a:  What  employee  is  more  experienced  than  educated  ? 

Example  b:  Who  is  older  than  young  ? 

Pattern  3:  [NP]i  [Rel.  Pronoun]  [BE- Verb]  [Adj  -{-compare]!  than 

Example:  The  terrorists  who  are  more  pro-Cuban  them 

[Adj],  [VP], 

pro-US  killed  the  Prime  Minister. 

Pattern  4:  [NP]i  [VP]  [Rel.  Pronoun]  [BE- Verb] 

Example:  The  terrorists  killed  the  Prime  Minister  who  is 

[Adj  -{-compare] i  than  [Adj]2. 
more  pro-Cuban  than  pro-US. 

Criteria:  Demonstrated  understanding  of  the  contrast  between  the  attributes  expressed  by 
[Adj  -{-compare] i  and  [Adj]2  for  the  entity  expressed  by  [NP]i- 

_ Score: _ 


1.2  Comparison  to  a  Lower  Degree 

A  comparative  expresses  a  comparison  or  contrast  between  two  entities  or  attributes  with 
respect  to  some  measure  on  a  scale,  which  may  be  numerical  or  non-numerical.  In  compar¬ 
atives  to  a  lower  degree,  the  measurement  of  the  first  entity  or  attribute  is  less  than  that  of 
the  second. 

A  comparison  to  a  lower  degree  is  formed  by  replacing  “[Adj  -{-compare]  than ”  in  the  standard 
comparative  adjective  pattern  in  the  beginning  of  the  section  titled  “Comparative  Adjectives”, 
with  uless  [Adj]  ihanr'. 

Perform  the  following  tests  with  comparison  to  a  lower  degree.  The  following  four  subsections 
test  comparison  to  a  lower  degree  with  no  gapping,  with  adjective  gapping,  with  copular  and 
adjective  gapping,  and  with  noun  phrase  and  copular  gapping,  respectively.  This  section  is 
parallel  to  Section  1.1. 

In  contrast  to  section  1.1,  the  patterns  are  not  repeated  in  each  subsection.  As  the  sections 
are  parallel,  we  rely  on  the  evaluators  to  follow  the  instructions  in  a  manner  similar  to  that 
done  in  1.1. 

LOWER  COMPARATIVE  ADJECTIVE  PATTERNS: 

Pattern  1:  List  [NP]i  [WH-Word]  [BE- Verb]  less  [Adj]i  than 

Example:  List  an  employee  who  is  less  experienced  than 
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[NP]a  [BE- Verb]  [Adj]2. 

the  Supervisor  of  Department  77  is  experienced. 

Pattern  2:  [WH-Word]  ([NP],)  [BE- Verb]  less  [Adfathan  [NP]2 

Example  a:  What  employee  is  less  experienced  than  Jane  Doe 

Example  b:  Who  is  less  educated  than  John  Smith 

[BE- Verb]  [Adj]2  ? 
is  experienced  ? 

is  educated  ? 


Pattern  3: 
Example: 


[NP], 

The  terrorists 

[NP]2 

ARENA 


[Rel.  Pronoun]  [BE- Verb]  less  [Adj]i  than 

who  are  less  anticommunist  than 


[BE- Verb]  [Adj]2 
is  anticommunist 


(VP]. 

killed  the  Prime  Minister. 


Pattern  4:  [NP]i  [VP]  [Rel.  Pronoun]  [BE- Verb] 

Example:  The  terrorists  killed  the  Prime  Minister  who  is 

less  [Adj]i  than  [NP]2  [BE- Verb]  [Adj]2. 

less  pro- Cuban  than  he  is  pro-US. 

Pattern  5:  [NP]!  [BE- Verb]  less  [Adj]i  than  [NP]2 

Example:  (Preliminary  Sentence:  Guerrillas  attacked  the  presidential  farm.) 

The  attack  was  less  deadly  than  the  previous  attack 

[BE-Verb]  [Adj]2. 
was  deadly. 

1.2.1  No  Gapping 

Test  a  comparative  adjective  to  a  lower  degree  with  no  gapping,  using  the  LOWER  COM¬ 
PARATIVE  ADJECTIVE  PATTERNS. 


Eg,  List  a  manager  vho  is  less  experienced  than  the  Supervisor 
of  Department  77  is  experienced. 

Eg,  Who  is  less  educated  than  John  Smith  is  educated? 

Eg,  The  terrorists  who  are  less  anticommunist  than  ARENA  is  anti¬ 
communist  killed  the  Prime  Minister. 

Eg,  The  terrorists  killed  the  Prime  Minister  who  is  less  Marxist 
then  he  is  Maoist. 

Eg,  (Preliminary  Sentence:  Guerrillas  attacked  the  presidential  farm.) 
The  attack  was  less  deadly  than  the  previous  attack  was  deadly. 
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Criteria:  Demonstrated  understanding  of  either 

•  the  contrast  between  the  two  entities  expressed  by  [NP]j  and  [NP]2  with  respect  to 

the  attribute  or  attributes  expressed  by  [Adj  +compare]i  and  [Adj]2. 

or 

•  the  contrast  between  the  two  attributes,  expressed  by  [Adj  +compare]i  and  [Adj]2, 

of  one  entity  expressed  by  both  [NP]i  and  [NP]2. 

_  Srnw _ 


1.2.2  Adjective  Gapping 

Test  a  comparative  adjective  to  a  lower  degree  with  adjective  gapping  by  omitting  the  [Adj]2 
from  the  LOWER  COMPARATIVE  ADJECTIVE  PATTERNS. 

Eg.  List  a  manager  who  is  less  experienced  than  the  Supervisor 
of  Department  77  is. 

Eg,  Who  is  less  educated  than  John  Smith  is? 

Eg,  The  terrorists  who  are  less  anticommunist  than  ARENA  is,  killed 
the  Prime  Minister. 

Eg.  The  terrorists  killed  the  Prime  Minister  who  is  less  Marxist 
than  they  are. 

Eg,  (Preliminary  Sentence:  Guerrillas  attacked  the  presidential  farm.) 
The  attack  was  less  deadly  than  the  previous  attack  was. 

Criteria:  Demonstrated  understanding  of  the  contrast  between  the  two  entities  expressed  by 
[NP]i  and  [NP]2  with  respect  to  the  attribute  expressed  by  [Adj  +compare]. 

_ Score: _ 


1.2.3  Copular  and  Adjective  Gapping 

Test  a  comparative  adjective  to  a  lower  degree  with  copular  and  adjective  gapping  by  omit¬ 
ting  the  [BE- Verb]  and  the  [Adj]2  from  the  LOWER  COMPARATIVE  ADJECTIVE  PAT¬ 
TERNS. 

Eg,  List  a  manager  who  is  less  experienced  than  the  Supervisor 
of  Department  77. 

Eg,  Who  is  less  educated  than  John  Smith? 

Eg,  The  terrorists  who  are  less  anticommunist  them  ARENA,  killed 
the  Prime  Minister. 

Eg,  The  terrorists  killed  the  Prime  Minister  who  is  less  Marxist 
than  they. 

Eg,  (Preliminary  Sentence:  Guerrillas  attacked  the  presidential  farm.) 
The  attack  was  less  deadly  than  the  previous  attack. 
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Criteria:  Demonstrated  understanding  of  the  contrast  between  the  two  entities  expressed  by 
[NP]!  and  [NP]2  with  respect  to  the  attribute  expressed  by  [Adj  +compare]. 

_ Score:  _ 


1.2.4  Noun  Phrase  and  Copular  Gapping  (Pseudo-Comparatives) 

Test  a  comparative  adjective  to  a  lower  degree  with  noun  phrase  and  copular  gapping  by 
omitting  the  [NP]2  and  [BE- Verb]  from  the  LOWER  COMPARATIVE  ADJECTIVE  PAT¬ 
TERNS.  The  adjectives  must  be  different. 

Eg.  List  a  manager  who  is  less  experienced  than  educated. 

Eg,  The  terrorists  who  are  less  anticommunist  than  antifascist , 
killed  the  Prime  Minister. 

Eg,  The  terrorists  killed  the  Prime  Minister  who  is  less  Marxist 
them  Maoist. 

Criteria:  Demonstrated  understanding  of  the  contrast  between  the  attributes  expressed  by 
[Adj  -J-compare]!  and  [Adj]2  for  the  entity  expressed  by  [NP]i. 

_ Score: _ 


1.3  Comparisons  To  the  Same  Degree 
1.3.1  In  Assertive  Contexts 

A  comparative  expresses  a  comparison  or  contrast  between  two  entities  or  attributes  with 
respect  to  some  measure  on  a  scale,  which  may  be  numerical  or  non-numerical.  In  a  com¬ 
parison  to  the  same  degree  in  an  assertive  context,  the  measurement  of  the  first  entity  or 
attribute  is  the  same  as  that  of  the  second. 

A  comparison  to  the  same  degree  in  an  assertive  context  is  formed  by  replacing  “[Adj  +com- 
pare]  than ”  in  the  standard  comparative  adjective  pattern  in  the  beginning  of  the  section 
titled  “Comparative  Adjectives”,  with  “ as  [Adj]  as”. 

Perform  the  following  tests  with  comparison  to  the  same  degree  in  an  assertive  context. 
The  following  four  subsections  test  comparison  to  the  same  degree  with  no  gapping,  with 
adjective  gapping,  with  copular  and  adjective  gapping,  and  with  noun  phrase  and  copular 
gapping,  respectively.  This  section  is  parallel  to  Section  1.1. 

SAME  DEGREE  COMPARATIVE  ADJECTIVE  PATTERNS. 

Pattern  1:  List  [NP]i  [WH-Word]  [BE- Verb]  as  [Adj]i  as 

Example:  List  an  employee  who  is  as  experienced  as 
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[NP]2  [BE-Verb]  [Adj]2. 

the  Supervisor  of  Department  77  is  experienced. 

Pattern  2:  [WH-Word]  ([NP^)  [BE-Verb]  as  [Adj],  as  [NP]2 

Example  a:  What  employee  is  as  experienced  as  Jane  Doe 

Example  b:  Who  is  as  old  as  John  Smith 

[BE-Verb]  [Adj]2  ? 
is  experienced  ? 

is  old  ? 

Pattern  3:  [NP]j  [Rel.  Pronoun]  [BE-Verb]  as  [Adj]i  as 

Example:  1  he  terrorists  who  are  as  anticommunist  as 

[NP]2  [BE-Verb]  [Adj]2  [VP]. 

ARENA  is  anticommunist  killed  the  Prime  Minister. 

Pattern  4:  [NP]i  [VP]  [Rel.  Pronoun]  [BE-Verb] 

Example:  The  terrorists  killed  the  Prime  Minister  who  is 

as  [Adj]!  as  [NP]2  [BE-Verb]  [Adj]2. 
as  pro-Cuban  as  he  is  pro-US 

Pattern  5:  [NP]i  [BE-Verb]  as  [Adj]j  as  [NP]2 

Example:  (Preliminary  Sentence:  Guerrillas  attacked  the  presidential  farm.) 

The  attack  was  as  deadly  as  the  previous  attack 

[BE-Verb]  [Adj]2. 
was  deadly. 

1.3. 1.1  No  Gapping 

Test  a  comparative  adjective  to  the  same  degree  with  no  gapping,  using  the  SAME  DEGREE 
COMPARATIVE  ADJECTIVE  PATTERNS. 

Eg,  What  employee  is  as  old  as  the  supervisor  of  department  17 
is  old? 

Eg,  The  terrorists  who  are  as  anticommunist  as  ARENA  is  anti¬ 
communist  killed  the  Prime  Minister. 

Eg,  The  terrorists  killed  the  Prime  Minister  who  is  as  Marxist 
as  he  is  Maoist. 

Eg,  (Preliminary  Sentence:  Guerrillas  attacked  the  presidential  farm.) 
The  attack  was  as  deadly  as  the  previous  attack  was  deadly. 

Criteria:  Demonstrated  understanding  of  either 
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•  the  contrast  between  the  two  entities  expressed  by  [NP]X  and  [NP]2  with  respect  to 

the  attribute  or  attributes  expressed  by  [Adj  +compare]x  and  [Adj]2. 

or 

•  the  contrast  between  the  two  attributes,  expressed  by  [Adj  +compare]x  and  [Adj]2, 

of  one  entity  expressed  by  both  [NP]X  and  [NP]2. 

_ _ Score: _ 


1.3. 1.2  Adjective  Gapping 

Test  a  comparative  adjective  to  the  same  degree  with  adjective  gapping  by  omitting  the 
[Adj]2  from  the  SAME  DEGREE  COMPARATIVE  ADJECTIVE  PATTERNS. 

Eg,  What  employee  is  as  old  as  the  supervisor  of  department  17 
is? 

Eg,  The  terrorists  who  are  as  anticommunist  as  ARENA  is,  killed 
the  Prime  Minister. 

Eg,  The  terrorists  killed  the  Prime  Minister  who  is  as  Marxist 
as  they  are. 

Eg,  (Preliminary  Sentence:  Guerrillas  attacked  the  presidential  farm.) 
The  attack  was  as  deadly  as  the  previous  attack  was. 

Criteria:  Demonstrated  understanding  of  the  contrast  between  the  two  entities  expressed  by 
[NP]X  and  [NP]2  with  respect  to  the  attribute  expressed  by  [Adj  +compahe], 

_ Score: _ 


1.3. 1.3  Copular  and  Adjective  Gapping 

Test  a  comparative  adjective  to  the  same  degree  with  copular  and  adjective  gapping  by  omit¬ 
ting  the  [BE- Verb]  and  the  [Adj]2  from  the  SAME  DEGREE  COMPARATIVE  ADJECTIVE 
PATTERNS. 

Eg,  What  employee  is  as  old  as  the  supervisor  of  department  17? 

Eg,  The  terrorists  who  are  as  anticommunist  as  ARENA,  killed  the 
Prime  Minister. 

Eg,  The  terrorists  killed  the  Prime  Minister  who  is  as  Marxist  as 
they. 

Eg,  (Preliminary  Sentence:  Guerrillas  attacked  the  presidential  farm.) 
The  attack  was  as  deadly  as  the  previous  attack. 

Criteria:  Demonstrated  understanding  of  the  contrast  between  the  two  entities  expressed  by 
[NP]X  and  [NP]2  with  respect  to  the  attribute  expressed  by  [Adj  Icompare], 
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Score:. 


1.3. 1.4  Noun  Phrase  and  Copular  Gapping  (Pseudo-Comparatives) 

Test  a  comparative  adjective  to  the  same  degree  with  noun  phrase  and  copular  gapping  by 
omitting  the  [NP]2  and  the  (BE- Verb]  from  the  SAME  DEGREE  COMPARATIVE  ADJEC¬ 
TIVE  PATTERNS.  The  adjectives  must  be  different. 


Eg,  What  employee  is  as  experienced  as  educated? 

Eg,  The  terrorists  who  are  as  anticommunist  as  antifascist,  killed 
the  Prime  Minister. 

Eg,  The  terrorists  killed  the  Prime  Minister  who  is  as  Marxist  as 
Maoist . 


Criteria:  Demonstrated  understanding  of  the  contrast  between  the  attributes  expressed  by 
[Adj  +compare]x  and  [Adj]2  for  the  entity  expresses  by  [NP]i. 

_ — _ _ _ Score: _ 


1.3.2  In  Non- Assertive  Contexts 

A  comparative  expresses  a  comparison  or  contrast  between  two  entities  or  attributes  with 
respect  to  some  measure  on  a  scale,  which  may  be  numerical  or  non- numerical.  In  compar¬ 
atives  to  the  same  degree  in  non-assertive  contexts,  the  measurement  of  the  first  entity  or 
attribute  is  not  the  same  as  that  of  the  second. 

An  adjective  comparison  to  the  same  degree  in  non-assertive  contexts,  i.e.,  in  negative  state¬ 
ments  or  questions,  is  formed  by  replacing  “[Adj  +compare]  than "  in  the  standard  compara¬ 
tive  adjective  pattern  in  the  beginning  of  the  section  titled  “Comparative  Adjectives” ,  with 
unot  so  [Adj]  as"  or  unot  as  [Adj]  as".  For  example, 

Richard  is  not  so  rich  as  he’d  like. 

Who  is  not  as  nice  as  Jenny? 

Determine  which  of  the  two  forms  (with  either  “as”  or  “so”),  if  either,  is  accepted  by  the 
system  being  tested,  and  use  that  form  for  the  test  questions. 

The  following  four  subsections  test  comparison  to  the  same  degree  with  no  gapping,  with 
adjective  gapping,  with  copular  and  adjective  gapping,  and  with  noun  phrase  and  copular 
gapping,  respectively.  This  section  is  parallel  to  Section  1.1. 

NON-ASSERTIVE  COMPARATIVE  ADJECTIVE  PATTERNS. 
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Pattern  1:  List  [NP]j  [WH-Word]  [BE- Verb]  not  so/as  [Adj]i  as 

Example:  List  an  employee  who  is  not  as  experienced  as 

[NP]2  [BE-Verb]  [Adj]2. 

the  Supervisor  of  Department  77  is  experienced. 


Pattern  2: 

[WH-Word] 

<[NP1.) 

[BE-V 

erb]  not  so/as  [Adj]i  as 

[NP]2 

Example  a: 

What 

employee 

is 

not  so  experienced  as 

Jane  Doe 

Example  b: 

Who 

is 

not  as  old  as 

John  Smith 

[BE-Verb] 

[Adj]a  ? 

is 

experienced  ? 

is 

old  ? 

Pattern  3: 

[NPh 

[Rel.  Pronoun] 

[BE-Verb] 

not  so/as  [Adj] 

i  as 

Example: 

The  terrorists 

who 

are 

not  so  anticommunist  as 

[NP]2 

[BE-Verb] 

[Adj]2 

[VP]. 

ARENA 

is 

anticommunist 

killed  the  Prime  Minister. 

Pattern  4: 

[NP]i 

[VP] 

[Rel.  Pronoun] 

[BE-Verb] 

Example: 

The  terrorists 

killed  the  Prime  Minister 

who 

is 

not  so/as 

[Adj]x  as 

[NP]a 

[BE-Verb]  [Adj]2. 

not  as  pro- Cuban  as 

he 

is 

pro-US. 

Pattern  5:  [NP]i  [BE-Verb]  not  so/as  [Adj]i  as  [NP]2 

Example:  (Preliminary  Sentence:  Guerrillas  attacked  the  presidential  farm.) 

The  attack  was  not  so  deadly  as  the  previous  attack 

[BE-Verb]  [Adj]2. 
was  deadly. 


1.3. 2.1  No  Gapping 

Test  a  comparative  adjective  to  the  same  degree  with  no  gapping,  using  the  NON-ASSERTIVE 
COMPARATIVE  ADJECTIVE  PATTERNS. 

Eg.  List  an  employee  who  is  not  as  experienced  as  the  Supervisor 
of  Department  77  is  experienced. 

Eg,  The  terrorists  vho  are  not  so  anticommunist  as  ARENA  is  anti¬ 
communist  killed  the  Prime  Minister. 

Eg,  The  terrorists  killed  the  Prime  Minister  who  is  not  as  Marxist 
as  he  is  Maoist . 

Eg,  (Preliminary  Sentence:  Guerrillas  attacked  the  presidential  farm.) 
The  attack  was  not  as  deadly  as  the  previous  attack  was  deadly. 
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Criteria:  Demonstrated  nderstanding  of  either 

•  the  contrast  between  the  two  entities  expressed  by  [NP]i  and  [NP]2  with  respect  to 

the  attribute  or  attributes  expressed  by  [Adj  +comparf]j  and  [Adj]2. 

or 

•  the  contrast  between  the  two  attributes,  expressed  by  [Adj  -f  com  pare]  i  and  [Adj]2, 

of  one  entity  expressed  by  both  [NP]i  and  [NP]2. 

_ Score: _ 


1. 3.2.2  Adjective  Gapping 

Test  a  comparative  adjective  to  the  same  degree  with  adjective  gapping  by  omitting  the 
[Adj]2  from  the  NON-ASSERTIVE  COMPARATIVE  ADJECTIVE  PATTERNS. 


Eg,  List  an  employee  who  is  not  so  experienced  as  the  Supervisor 
of  Department  77  is. 

Eg,  The  terrorists  who  are  not  so  anticommunist  as  ARENA  is,  killed 
the  Prime  Minister. 

Eg,  The  terrorists  killed  the  Prime  Minister  who  is  not  as  Marxist 
as  they  are. 

Eg,  (Preliminary  Sentence:  Guerrillas  attacked  the  presidential  farm.) 
The  attack  was  not  as  deadly  as  the  previous  attack  was. 

Criteria:  Demonstrated  understanding  of  the  contrast  between  the  two  entities  expressed  by 
[NP]i  and  [NP];  with  resppc*  to  the  attribute  expressed  by  [Adj  Icompare], 

_ Score: _ 


1.3. 2. 3  Copular  and  Adjective  Gapping 

Test  a  comparative  adjective  to  the  same  degree  with  copular  and  adjective  gapping  by 
omitting  the  [BE- Verb]  and  the  [Adj]2  from  the  NON-ASSERTIVE  COMPARATIVE  AD¬ 
JECTIVE  PATTERNS. 

Eg,  List  an  employee  who  is  not  as  experienced  as  the  Supervisor 
of  Department  77. 

Eg,  The  terrorists  who  are  not  so  anticommunist  as  ARENA,  killed 
the  Prime  Minister. 

Eg,  The  terrorists  killed  the  Prime  Minister  who  is  not  as  Marxist 
as  they. 

Eg,  (Preliminary  Sentence:  Guerrillas  attacked  the  presidential  farm.) 
The  attack  was  not  as  deadly  as  the  previous  attack. 
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Criteria:  Demonstrated  understanding  of  the  contrast  between  the  two  entities  expressed  by 
[NP]!  and  [NP]2  with  respect  to  the  attribute  expressed  by  [Adj  Icompare]. 

_ Score: _ 


1.3. 2.4  Noun  Phrase  and  Copular  Gapping  (Pseudo-Comparatives) 

Test  a  comparative  adjective  to  the  same  degree  with  noun  phrase  and  copular  gapping 
by  omitting  the  [NP]2  and  the  [BE- Verb]  from  the  NON- ASSERTIVE  COMPARATIVE 
ADJECTIVE  PATTERNS.  The  adjectives  must  be  different. 

Eg,  List  an  employee  vho  is  not  so  experienced  as  educated. 

Eg,  The  terrorists  who  are  not  so  anticommunist  as  antifascist, 
killed  the  Prime  Minister. 

Eg,  The  terrorists  killed  the  Prime  Minister  who  is  not  as  Marxist 
as  Maoist. 

Criteria:  Demonstrated  understanding  of  the  contrast  between  the  two  attributes  expressed 
by  [Adj  +compare]i  and  [Adj]2. 

_ Score: _ 


1.4  Comparison  to  a  Constant 

A  constant  is  an  individual  value  from  the  scale  used  to  measure  an  attribute.  For  example: 

3.5  years  is  a  constant  from  the  scale  used  to  measure  time  in  years;  6.75  in  is  a  constant 
from  the  scale  used  to  measure  a  linear  dimension  in  inches;  excellent  is  a  constant  from  the 
scale  used  to  measure  quality  ( excellent ,  good,  fair,  poor). 

A  comparison  to  a  constant  defines  the  measure  of  an  attribute  by  comparing  it  to  a  constant 
measure.  For  example,  the  phrase  tiller  than  6  feet  expresses  an  attribute  defined  by  a 
comparison  to  the  constant  6  feet. 

Comparisons  to  a  constant  can  be  done  to  a  higher,  lower,  or  equal  degree.  The  examples 
given  here  are  to  a  higher  degree. 

We  use  [NP  +constant]  to  denote  a  noun  phrase  with  a  constant  functioning  either  as  the 
head  or  as  a  determinative. 

Pattern  1:  List  [NP]i  [WH-Word]  [BE- Verb]  [Adj  +compare]i  than 

Example:  List  an  employee  who  is  more  experienced  than 

[NP  -(-constant]. 
five  years. 


VII- 15 


Pa»s  n  2: 

[WH-Word]  ([NP]i) 

[BE-Verb] 

[Adj  +compare]i  than 

[NP  -{-constant]? 

Example: 

Who 

is 

older 

than 

50? 

Pattern  3: 

(NP]i 

[Rel.  Pronoun]  [BE)- Verb] 

[Adj  +compare]i  than 

Example: 

The  Prime  Minister 

who 

is 

older 

than 

[NP  -{-constant]  [VP]. 

70  was  killed. 


Pattern  4:  [NP]i  [VP]  [Rel.  Pronoun]  [BE- Verb] 

Example:  The  terrorists  killed  the  Prime  Minister  who  is 

[Adj  -{-compare]  than  [NP  -f constant]. 
older  than  60  years. 


Pattern  5:  [NP]x  [BE)- Verb] 

Example:  (Preliminary  Sentence:  Guerrillas  attacked  government  officials.) 

The  number  of  government  officials  who  were  attacked  is 

[Adj  +compare]  than  [NP  -{-constant]. 
greater  than  5. 

Criteria:  Demonstrated  understanding  of  the  measure  of  the  attribute  expressed  by  the 
comparison  to  a  constant. 

_  Score:  _ 


1.5  Comparative  Phrases  Used  as  Pre-Modifiers  Within  a  Noun  Phrase 
A  [Comparative  Phrase]  may  compare: 

•  two  adjectives: 

more  experienced  than  educated 

•  an  adjective  with  a  constant: 

older  than  50. 

•  an  adjective  with  a  noun  phrase: 

as  loud  as  a  siren 

Comparative  phrases  may  be  to  a  higher,  a  lower,  or  the  same  degree. 

The  following  sentences  use  a  comparative  phrase  as  a  pre-modifier  in  a  noun  phrase.  Recall 
that  the  pre-modifier  appears  before  the  head  in  a  NP. 
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Pattern:  List  ([DET])  [Comparative  Phrase]  [NP]. 

Pattern:  [WH-Word]  [Comparative  Phrase]  [NP]  [VP]  ? 

Example:  What  older  than  65  years  employee  earns  $25,000? 

Pattern:  [NP]  [Verb]  ([DET])  [Comparative  Phrase]  [NP]. 

Example:  The  terrorists  killed  the  more  Marxist  than  Maoist  Prime  Minister. 

Pattern:  ([DET])  [Comparative  Phrase]  [NP]  [VP]. 

Example  1:  The  more  Marxist  than  Maoist  Prime  Minister  was  killed  by  terrorists. 

Example  2:  (Preliminary  Sentence:  Guerrillas  attacked  a  farm  on  January  2.) 

The  more  damaged  than  destroyed  farm  was  attacked  again 

on  February  10. 


Criteria:  Demonstrated  understanding  of 

•  the  comparison  expressed  by  the  comparative  phrase 

•  the  attribute  or  quality  imposed  on  the  entity  expressed  by  the  noun  phrase 

which  the  comparative  phrase  premodifies. 

_ _ _ Score:. 


2.  Superlatives 

A  superlative  (  “youngest”,  “most  educated”,  “least  qualified”)  indicates  that  the  entity  being 
modified  is  at  one  of  the  extreme  ends  of  the  scale  with  respect  to  the  property  being 
attributed  to  it.  For  example,  “the  youngest  employee”  refers  to  the  employee  that  is  at  the 
lowest  end  of  the  age  scale. 

2.1  Superlatives  to  a  High  Degree 

A  superlative  to  a  high  degree  indicates  that  the  entity  to  whom  the  superlative  applies  has 
the  greatest  amount  of  an  attribute  or  comes  the  closest  to  the  “high”  endpoint  on  the  scale 
associated  with  an  attribute  about  which  the  comparison  is  made  (e.g.,  “ the  most  educated 
scholar”). 

Superlatives  to  a  high  degree  are  formed: 

•  by  adding  an  - est ,  with  an  associated  spelling  change  if  needed: 

big  — ►  biggest 

•  by  using  the  word  most  before  the  adjective: 

beautiful  - — ♦  most  beautiful 
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In  forming  superlatives  to  a  high  degree,  test  both  the  most  and  the  - est  inflection.  The 
following  suggested  patterns  capture  the  structure  which  we  are  testing. 

Pattern:  List  ([DET])  [Adj  +super]  [NP]. 

Example:  List  the  oldest  salesperson. 

Pattern:  What  [NP]  [BE- Verb]  ([DET])  [Adj  +  so  per]  ? 

Pattern:  [WH-Word]  [BE- Verb]  ([DET])  [Adj  +super]  [NP]  ? 

Example:  Who  is  the  most  educated  employee  in  the  NY  branch? 

Pattern:  [NP]  [Verb]  ([DET])  [Adj  4-super]  [NP]. 

Example:  Terrorists  killed  the  most  fascist  Government  official. 

Pattern:  ([DET])  [Adj  -(-super]  [NP]  [VP], 

Example  1 :  The  most  fascist  Government  official  was  killed  by  terrorists. 

Example  2:  (Preliminary  Sentence:  Guerrillas  attacked  farms.) 

The  most  damaged  farm  is  the  Santo  Tomas  farm. 

With  “most”: 

Criteria:  Demonstrated  understanding  that  the  superlative  to  a  high  degree  formed  with  an 
“-est”  suffix  expresses  an  attribute  at  the  “high”  endpoint  of  the  scale  associated  with  that 
attribute. 

_ Score: _ 

With  “-est”: 

Criteria:  Demonstrated  understanding  that  the  superlative  to  a  high  degree  formed  with  the 
word  “most”  expresses  an  attribute  at  the  “high”  endpoint  of  the  scale  associated  with  that 
attribute. 

- - Score: _ 


2.2  Superlatives  to  a  Low  Degree 

A  superlative  to  a  low  degree  indicates  that  the  entity  to  whom  the  superlative  applies  has 
the  least  amount  of  an  attribute  or  comes  the  closest  to  the  “low”  endpoint  on  the  scale 
associated  with  an  attribute  about  which  the  comparison  is  made  (e.g.,  “ the  least  qualified 
employee”). 

Superlatives  to  a  low  degree  are  formed: 

•  with  the  word  least: 
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qualified  — >  least  qualified 

The  following  are  suggested  sentence  patterns: 

Pattern:  List  ([DET])  least  [Adj]  [NP]. 

Example:  List  the  least  educated  salesperson. 


Pattern: 

What  [NP] 

[BE- Verb] 

([DET]) 

least 

[Adj]  ? 

Pattern: 

[WH-Word] 

[BE-Verb] 

([DET]) 

least 

[Adj] 

[NP]  ? 

Example: 

Who 

is 

the 

least 

educated 

employee  in  the  NY  branch? 

Pattern:  [NP]  [Verb]  ([DET])  least  [Adj]  [NP]. 

Example:  Terrorists  killed  the  least  fascist  Government  official. 

Pattern:  ([DET])  least  [Adj]  [NP]  [VP]. 

Example  1 :  The  least  fascist  Government  official  was  killed  by  terrorists. 

Example  2:  (Preliminary  Sentence:  Guerrillas  attacked  farms.) 

The  least  damaged  farm  is  the  Santo  Tomas  farm. 

Criteria:  Demonstrated  understanding  that  the  superlative  to  a  low  degree  expresses  an 
attribute  at  the  “low”  endpoint  of  the  scale  associated  with  that  attribute. 

_ Score: _ 


3.  Comparative  Adverbial  Phrase 

A  comparative  expresses  a  comparison  or  contrast  between  two  events,  entities,  or  attributes 
with  respect  to  some  measure  on  a  scale,  which  may  be  numerical  (age,  height,  speed)  or 
non-numerical  (friendliness,  perseverance).  In  the  sentence: 

Jones  sold  printers  more  often  than  Smith. 

the  comparison  is  between  the  actions  ( selling  printers)  of  two  entities  Jones  and  Smith  with 
respect  to  time,  a  numerical  measure. 

Typically,  in  interrogatives,  the  comparison  is  between  a  known  and  an  unknown  entity.  For 
example,  in  the  question: 

Who  is  older  than  John  ? 

the  comparison  is  between  John  and  an  unknown  someone  (expressed  by  the  WH-vroid 
“who”). 
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The  standard  comparative  adverb  pattern  is: 

[NP]i  [Verb]  [Adverb  +compare]i  than  [NP]2  (Verb]2 
3.1  Comparison  to  a  Higher  Degree 

In  comparatives  to  a  higher  degree,  the  measurement  of  the  first  event,  entity  or  attribute 
is  greater  than  that  of  the  second. 

Adverb  comparatives  to  a  higher  degree  are  formed: 

•  by  adding  an  -er: 

fast  — ►  faster 

•  by  dropping  the  -ly  and  adding  -er: 

quickly  — ►  quicker 

•  by  using  the  word  more  before  the  adverb: 

beautifully  — ►  more  beautifully 

Caution:  “ More  ”  can  be  used  as  a  quantifier  in  a  noun  phrase  as  well  as  part  of  a  comparative 
adverb  group.  “Much”  can  also  be  used  as  a  quantifier  in  a  noun  phrase,  as  well  as  a 
comparative  adverb.  In  this  section,  avoid  the  use  of  “more”  or  “much”  as  quantifiers. 

MAIN  COMPARATIVE  ADVERB  PATTERNS: 


Pattern:  List  [NP^  [Rel  Pronoun] 

[VP], 

[Adverb  4-compare] 

than  [NP]2  [VP] 

Pattern:  [Wh-Word]  ([NP],) 

[VP], 

[Adverb  4-compare]  than 

Example  1:  Which  salesperson 

sells  printers  more  often 

than 

Example  2:  Who 

advanced  more  quickly 

than 

[NP]2  [VPj2  ? 

Smith  sells  CPUs? 

John  Smith  advanced? 

Pattern:  [NP]i 

[VP], 

[Adverb  4-compare] 

Example  1:  Shining  Path 

murdered  civilians 

more  ruthlessly 

Example  2:  Government  officials  in  Honduras 

are  kidnapped 

more  often 

than  [NP]2 

[VP]2- 

than  ARENA 

assassinated  government  officials. 

than  government  officials  in 

Guatemala  are  killed. 

3.1.1  No  Gapping 
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Test  a  comparative  adverb  to  a  higher  degree  with  no  gapping,  using  the  MAIN  COMPAR¬ 
ATIVE  ADVERB  PATTERNS. 


Eg,  Which  employee  vas  hired  earlier  than  Smith  was  hired? 

Eg,  Which  salesperson  sells  printers  more  often  than  Smith  sells 
CPUs? 

Eg,  Who  advanced  more  quickly  than  John  Smith  advanced? 

Eg,  Shining  Path  murdered  civilians  more  ruthlessly  than  ARENA 
assassinated  government  officials. 

Eg,  Government  officials  in  Honduras  are  kidnapped  more  often  than 
government  officials  in  Guatemala  are  killed. 


Criteria:  Demonstrated  understanding  of  the  contrast  between  the  two  events  expressed  by 
the  subject-verb  phrase  pairs,  [NP]X  [VP]X  and  [NP]2  [VP]2,  with  respect  to  the  attribute 
expressed  by  [Adverb  +compare]. 

_ Score: _ 


3.1.2  Verb  phrase  gapping 

Test  a  comparative  adverb  to  a  higher  degree  with  verb  phrase  gapping  by  omitting  the 
[VP]2  from  the  MAIN  COMPARATIVE  ADVERB  PATTERNS. 


Pattern:  List  [NP]X  [Rel  Pronoun] 

(VP]i  [Adverb  +  compare] 

than  [NP]2 

Pattern:  [Wh-Word]  ([NP]X) 

[VP]j  [Adverb  -{-compare]  than 

Example  1:  Which  salesperson 

sells  printers  more  often 

than 

Example  2:  Who 

advanced  more  quickly 

than 

[NP]2? 

Smith? 

John  Smith? 

Pattern:  [NP]i 

[VP]. 

[Adverb  -{-compare] 

Example  1:  Shining  Path 

murdered  civilians 

more  ruthlessly 

Example  2:  Government  officials  in  Honduras  are  kidnapped 

more  often 

than  [NP]2. 
than  ARENA. 

than  government  officials  in  Guatemala. 


Criteria:  Demonstrated  understanding  of  the  contrast  between  the  event  expressed  by  the 
subject-verb  phrase  pair  [NP]X  [VP]X  and  the  event  expressed  by  the  subject  [NP]2  with 
an  omitted  (but  implied)  verb  phrase,  with  respect  to  the  attribute  expressed  by  [Adverb 

-f  compare]. 
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Score: 


3.1.3  Noun  phrase  gapping 


Test  a  comparative  adverb  to  a  higher  degree  with  noun  phrase  gapping  by  omitting  the 
[NP]2,  the  subject  of  the  second  clause,  from  the  MAIN  COMPARATIVE  ADVERB  PAT¬ 
TERNS. 

Pattern:  List  [NP]  [Rel  Pronoun]  [VP]i  [Adverb  -(-compare]  than  [VP]2. 

Pattern:  [Wh-Word]  ([NP])  [VP]i  [Adverb  -(-compare]  than 

Example  Which  salesperson  sells  printers  more  often  than 

[VP]2  ? 
sells  CPUs? 


Pattern: 
Example  1: 
Example  2: 


[NP]  [VP]j  [Adverb  -(-compare] 

Shining  Path  murdered  civilians  more  ruthlessly 

Government  officials  in  Honduras  are  kidnapped  more  often 

than  [VP]2. 

than  assassinated  government  officials, 
than  are  killed. 


Criteria:  Demonstrated  understanding  of  the  contrast  between  the  event  expressed  by  the 
subject-verb  phrase  pair  [NP]x  [VP]j  and  that  expressed  by  [VP]2  with  an  omitted  (but 
implied)  subject,  with  respect  to  the  attribute  expressed  by  [Adverb  -(-compare]. 

_ Score:  _ 


3.2  Comparison  to  a  Lower  Degree 

In  comparatives  to  a  lower  degree,  the  measurement  of  the  first  event,  entity  or  attribute  is 
less  than  that  of  the  second. 

Adverb  comparatives  to  a  lower  degree  are  formed: 

•  with  the  word  less  before  the  adverb. 
quickly  — *  less  quickly 

Perform  the  following  tests  with  comparison  to  a  lower  degree.  Replace  the  “[Adverb  -f  com¬ 
pare]  than ”  in  the  standard  comparative  adverb  pattern  in  the  beginning  of  the  section  titled 
“Comparative  Adverbs”,  with  uless  [Adverb]  than”.  The  following  three  subsections  test 
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comparison  to  a  lower  degree  with  no  gapping,  with  verb  phrase  gapping,  and  with  noun 
phrase  gapping,  respectively.  This  section  is  parallel  to  Section  3.1 

LOWER  COMPARATIVE  ADVERB  PATTERNS: 


Pattern:  List  [NP]i 

[Rel  Pronoun] 

[VP]i  less  [Adverb]  than 

[NP], 

[VP]2. 

Pattern: 

[Wh-Word] 

«NP],) 

[VP], 

less  [Adverb] 

than 

[NP], 

Example  1 :  Which 

salesperson 

sells  printers 

less  often 

than 

Smith 

Example  2: 

Who 

advanced 

less  quickly 

than 

John  Smith 

[VP]2  ? 
sells  CPUs? 
advanced? 


Pattern: 

[NP], 

[VP], 

less  [Adverb] 

than 

Example  1: 

Shining  Path 

murdered  civilians 

less  ruthlessly 

than 

Example  2: 

Government  officials  in  Honduras 

[NP], 

are  kidnapped 

[VP],. 

less  often 

than 

ARENA  assassinated  government  officials, 

government  officials  in  Guatemala  are  killed. 

3.2.1  No  Gapping 

Test  a  comparative  adverb  to  a  lower  degree  with  no  gapping,  using  the  LOWER  COMPAR¬ 
ATIVE  ADVERB  PATTERNS. 

Eg,  Which  salesperson  sells  printers  less  often  than  Smith  sells 
CPUs? 

Eg,  Who  advanced  less  quickly  than  John  Smith  advanced? 

Eg,  Shining  Path  murdered  civilians  less  ruthlessly  them  ARENA 
assassinated  government  officials. 

Eg,  Government  officials  in  Honduras  are  kidnapped  less  often  than 
government  officials  in  Guatemala  are  killed. 

Criteria:  Demonstrated  understanding  of  the  contrast  between  the  two  events  expressed  by 
the  subject-verb  phrase  pairs,  [NP]i  [VP]i  and  [NP]2  [VP]2,  with  respect  to  the  attribute 
expressed  by  [Adverb  +  com  pare]. 

_ Score: _ 


3.2.2  Verb  Phrase  Gapping 

Test  a  comparative  adverb  to  a  lower  degree  with  verb  phrase  gapping  by  omitting  the  [VP]2 
from  the  LOWER  COMPARATIVE  ADVERB  PATTERNS. 
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Eg,  Which  salesperson  sells  printers  less  often  than  Smith? 

Eg,  Who  advanced  less  quickly  than  John  Smith? 

Eg,  Shining  Path  murdered  civilians  less  ruthlessly  than  ARENA. 

Eg,  Government  officials  in  Honduras  are  kidnapped  less  often  than 
government  officials  in  Guatemala. 

Criteria:  Demonstrated  understanding  of  the  contrast  between  the  event  expressed  by  the 
subject-verb  phrase  pair  [NP]j  [VP]!  and  that  expressed  by  the  subject  [NP]2  with  an  omitted 
(but  implied)  verb  phrase,  with  respect  to  the  attribute  expressed  by  [Adverb  +compare]. 

_ Score: _ 


3.2.3  Noun  Phrase  Gapping 

Test  a  comparative  adverb  to  a  lower  degree  with  noun  phrase  gapping  by  omitting  the 
[NP]2,  the  subject  of  the  second  clause,  from  the  LOWER  COMPARATIVE  ADVERB 
PATTERNS. 

Eg,  Which  salesperson  sells  printers  less  often  than  sells  CPUs? 

Eg,  Shining  Path  murdered  civilians  less  ruthlessly  than  assassinated 
government  officials. 

Eg,  Government  officials  in  Honduras  are  kidnapped  less  often  than 
are  killed. 

Criteria:  Demonstrated  understanding  of  the  contrast  between  the  event  expressed  by  the 
subject-verb  phrase  pair  [NP]i  [VP]x  and  that  expressed  by  [VP]2  with  an  omitted  (but 
implied)  subject,  with  respect  to  the  attribute  expressed  by  [Adverb  -(-compare]. 

_ Score: _ 


3.3  Comparisons  to  the  Same  Degree 

In  comparatives  to  the  same  degree,  the  measurement  of  the  first  event,  entity  or  attribute 
is  the  same  as  that  of  the  second. 

3.3.1  In  Assertive  Contexts 

An  adverb  comparison  to  the  same  degree  in  an  assertive  context,  is  formed  by  replacing 
“[Adverb  +compare]  than ”  in  the  standard  comparative  adverb  pattern  in  the  beginning  of 
the  section  titled  “Comparative  Adverbs”,  with  “as  [Adverb]  as”. 

Perform  the  following  tests  with  comparison  to  the  same  degree.  The  following  three  sub¬ 
sections  test  comparison  to  the  same  degree  with  no  gapping,  with  verb  phrase  gapping,  and 
with  noun  phrase  gapping,  respectively.  This  section  is  parallel  to  Section  3.1. 

SAME  DEGREE  COMPARATIVE  ADVERB  PATTERNS: 
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Pattern:  List  [NP]i  [Rel  Pronoun]  [VP]j  as  [Adverb]  as  [NP]2  [VP]2. 


Pattern:  [Wh-Word]  ((NP]t) 

Example  1:  Which  salesperson 

Example  2:  Who 

[VP]2  ? 
sells  CPUs? 
advanced? 


[VP], 

sells  printers 
advanced 


as  [Adverb]  as 
as  often  as 
as  quickly  as 


[NP], 

Smith 
John  Smith 


Pattern: 
Example  1: 
Example  2: 


[NP], 

Shining  Path 

Government  officials  in  Honduras 


[VP], 

murdered  civilians 
are  kidnapped 


as  [Adverb]  as 
as  ruthlessly  as 
as  often  as 


[NP],  [VP],. 

ARENA  assassinated  government  officials, 

government  officials  in  Guatemala  are  killed. 


3. 3. 1.1  No  Gapping 

Test  a  comparative  adverb  to  the  same  degree  with  no  gapping,  using  the  SAME  DEGREE 
COMPARATIVE  ADVERB  PATTERNS. 


Eg,  Which  salesperson  sells  printers  as  often  as  Smith  sells 
CPUs? 

Eg,  Who  advanced  as  quickly  as  John  Smith  advanced? 

Eg,  Shining  Path  murdered  civilians  as  ruthlessly  as  ARENA 
assassinated  government  officials. 

Eg,  Government  officials  in  Honduras  are  kidnapped  as  often  as 
government  officials  in  Guatemala  are  killed. 


Criteria:  Demonstrated  understanding  of  the  contrast  between  the  two  events  expressed  by 
the  subject-verb  phrase  pairs,  [NP]!  [VP]j  and  [NP]2  [VP]2,  with  respect  to  the  attribute 
expressed  by  [Adverb  +compare]. 

_ Score: _ 


3. 3. 1.2  Verb  Phrase  Gapping 

Test  a  comparative  adverb  to  the  same  degree  with  verb  phrase  gapping  by  omitting  the 
[VP]2  from  the  SAME  DEGREE  COMPARATIVE  ADVERB  PATTERNS. 


Eg,  Which  salesperson  sells  printers  as  often  as  Smith? 
Eg,  Who  advanced  as  quickly  as  John  Smith? 
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Eg,  Shining  Path  murdered  civilians  as  ruthlessly  as  ARENA. 

Eg.  Government  officials  in  Honduras  are  kidnapped  as  often  as 
government  officials  in  Guatemala. 

Criteria:  Demonstrated  understanding  of  the  contrast  between  the  event  expressed  by  the 
subject-verb  phrase  pair  [NP]i  [VP]!  and  that  expressed  by  the  subject  [NP]2  with  an  omitted 
(but  implied)  verb  phrase,  with  respect  to  the  attribute  expressed  by  [Adverb  +compahe]. 

_ Score: _ 


3.3. 1.3  Noun  Phrase  Gapping 

Test  a  comparative  adverb  to  the  same  degree  with  noun  phrase  gapping  by  omitting  the 
[NP]2,  the  subject  of  the  second  clause,  from  the  SAME  DEGREE  COMPARATIVE  AD¬ 
VERB  PATTERNS. 

Eg,  Which  salesperson  sells  printers  as  often  as  sells  CPUs? 

Eg,  Shining  Path  murdered  civilians  as  ruthlessly  as  assassinated 
government  officials. 

Eg,  Government  officials  in  Honduras  are  kidnapped  as  often  as  are 
killed. 

Criteria:  Demonstrated  understanding  of  the  contrast  between  the  event  expressed  by  the 
subject-verb  phrase  pair  [NP]i  [VP]i  and  that  expressed  by  [VP]2  with  an  omitted  (but 
implied)  subject,  with  respect  to  the  attribute  expressed  by  [Adverb  -(-compare]. 

_ Score: _ 


3.3.2  In  Non- Assertive  Contexts 

An  adverb  comparison  to  the  same  degree  in  non-assertive  contexts,  i.e.,  in  questions  or 
negative  statements,  is  formed  by  replacing  “[Adverb  +  compare]  than"  in  the  standard  com¬ 
parative  adverb  pattern  in  the  beginning  of  the  section  titled  “Comparative  Adverbs”,  with 
“not  so  [Adverb]  as"  or  “not  as  [Adverb]  as" .  Examples  are: 

Who  advanced  not  so  quickly  as  John  Smith  advanced? 

Smith  works  not  as  often  as  Jones. 

Determine  which  of  the  forms,  if  either,  is  accepted  by  the  system  being  tested,  and  use  that 
form  for  the  test  questions. 

The  following  three  subsections  test  comparison  to  the  same  degree  with  no  gapping,  with 
verb  phrase  gapping,  and  with  noun  phrase  gapping,  respectively.  This  section  is  parallel  to 
Section  3.1 
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NON-ASSERTIVE  COMPARATIVE  ADVERB  PATTERNS: 

Pattern:  List  [NP]i  [Rel  Pronoun]  [VP]i  not  so/as  [Adverb]  as  [NP]2  [VP]2. 


Pattern:  [Wh-Word]  ([NP]j)  [VP]i  not  so/as  [Adverb]  as  [NP]2 

Example  1:  Which  salesperson  sells  printers  not  so  often  as  Smith 

Example  2:  Who  advanced  not  as  quickly  as  John  Smith 

[VP]2  ? 
sells  CPUs? 
advanced? 

[NP]j  [VP]i  not  so/as  [Adverbjas 

Shining  Path  murdered  civilians  not  so  ruthlessly  as 

Government  officials  in  Honduras  are  kidnapped  not  as  often  as 

[NP]a  [VP]* 

ARENA  assassinated  government  officials, 

government  officials  in  Guatemala  are  killed. 

3. 3. 2.1  No  Gapping 

Test  a  comparative  adverb  to  the  same  degree  with  no  gapping,  using  the  NON-ASSERTIVE 
COMPARATIVE  ADVERB  PATTERNS. 

Eg,  Which  salesperson  sells  printers  not  as  often  as  Smith 
sells  CPUs? 

Eg,  Who  advanced  not  so  quickly  as  John  Smith  advanced? 

Eg,  Shining  Path  murdered  civilians  not  so  ruthlessly  as  ARENA 
assassinated  government  officials. 

Eg,  Government  officials  in  Honduras  are  kidnapped  not  as  often  as 
government  officials  in  Guatemala  are  killed. 

Criteria:  Demonstrated  understanding  of  the  contrast  between  the  two  events  expressed  by 
the  subject-verb  phrase  pairs,  [NP]i  [VP]i  and  [NP]2  [VP]2,  with  respect  to  the  attribute 
expressed  by  [Adverb  -{-compare]. 

- — . - - - Score: _ 

3.3. 2.2  Verb  Phrase  Gapping 

Test  a  comparative  adverb  to  the  same  degree  with  verb  phrase  gapping  by  omitting  the 
[VP]2  from  the  NON-ASSERTIVE  COMPARATIVE  ADVERB  PATTERNS. 

Eg,  Which  salesperson  sells  printers  not  as  often  as  Smith? 


Pattern: 
Example  1: 
Example  2: 
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Eg,  Who  advanced  not  so  quickly  as  John  Smith? 

Eg,  Shining  Path  murdered  civilians  not  so  ruthlessly  as  ARENA. 

Eg,  Government  officials  in  Honduras  are  kidnapped  not  as  often  as 
government  officials  in  Guatemala. 

Criteria:  Demonstrated  understanding  of  the  contrast  between  the  event  expressed  by  the 
subject-verb  phrase  pair  [NP]i  [VP]!  and  that  expressed  by  the  subject  [NP]2  with  an  omitted 
(but  implied)  verb  phrase,  with  respect  to  the  attribute  expressed  by  [Adverb  +compare]. 

— _ Score: _ 


3. 3. 2. 3  Noun  Phrase  Gapping 

Test  a  comparative  adverb  to  the  same  degree  with  noun  phrase  gapping  by  omitting  the 
[NP]2,  the  subject  of  the  second  clause,  from  the  NON-ASSERTIVE  COMPARATIVE  AD¬ 
VERB  PATTERNS. 

Eg,  Which  salesperson  sells  printers  not  as  often  as  sells 
CPUs? 

Eg,  Shining  Path  murdered  civilians  not  so  ruthlessly  as  assassinated 
government  officials. 

Eg,  Government  officials  in  Honduras  are  kidnapped  not  as  often  as 
are  killed. 

Criteria:  Demonstrated  understanding  of  the  contrast  between  the  event  expressed  by  the 
subject-verb  phrase  pair  [NP]i  [VP]!  and  that  expressed  by  [VP]2  with  an  omitted  (but 
implied)  subject,  with  respect  to  the  attribute  expressed  by  [Adverb  +compare]. 

— - - - - - Score: _ 


3.4  Comparisons  to  a  Constant 

A  constant  is  an  individual  value  from  the  scale  used  to  measure  an  attribute.  For  example: 

3.5  years  is  a  constant  from  the  scale  used  to  measure  time  in  years;  6.75  in  is  a  constant 
from  the  scale  used  to  measure  a  linear  dimension  in  inches;  excellent  is  a  constant  from  the 
scale  used  to  measure  quality  ( excellent ,  good,  fair,  poor). 

A  comparison  to  a  constant  defines  the  measure  of  an  attribute  by  comparing  it  to  a  constant 
measure.  For  example,  the  phrase  more  often  than  once  daily  expresses  an  attribute  defined 
by  a  comparison  to  the  constant  once  daily. 

Comparisons  to  a  constant  can  be  done  to  a  higher,  lower,  or  equal  degree.  The  examples 
given  here  are  to  a  higher  degree. 

We  use  [NP  +constant]  to  denote  a  noun  phrase  with  a  constant  functioning  either  a s  the 
head  or  as  a  determinative. 
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Pattern:  List  [NP]  [Rel  Pronoun]  [VP]  [Adverb  +compare]  than  [Constant] 

([Adverbial]). 

Pattern:  [Wh-Word]  ([NP])  [VP]  [Adverb  +compare]  than  [Constant]  ([Adverbial])? 

Example  Who  moved  more  often  than  three  times  in  1989  ? 

Pattern:  [NP]  [VP]  [Adverb  +compare]  than  [Constant] 

Example  Shining  Path  murdered  civilians  more  often  than  three  times 

([Adverbial]), 
last  year. 

Criteria:  Demonstrated  understanding  of  the  comparison  between  the  event  expressed  by 
the  [NP]  [VP]  pair  and  the  constant,  with  respect  to  the  attribute  expressed  by  [Adverb 
+compahe]. 

_ Score: _ 


4.  Superlative  Adverbs 

A  superlative  (  “most  often”,  “most  carefully”)  indicates  that  the  entity  being  modified  is  at 
one  of  the  extreme  ends  of  the  scale  with  respect  to  the  property  being  attributed  to  it.  For 
example,  in  the  sentence 

Of  all  the  salespersons,  Smith  sold  printers  most  easily. 

Smith  is  at  the  highest  end  of  the  scale  associated  with  easiness  with  respect  to  the  action 
of  selling  printers. 

4.1  Superlative  Adverbs  to  a  High  Degree 

A  superlative  to  a  high  degree  indicates  that  the  event  or  entity  to  whom  the  superlative 
applies  has  the  greatest  amount  of  an  attribute  or  comes  the  closest  to  the  “high”  endpoint 
on  the  scale  associated  with  an  attribute  about  which  the  comparison  is  made  (e.g.,  “most 
easily  sold”). 

Superlative  adverbs  to  a  high  degree  are  formed: 

•  by  adding  an  -est: 

fast  — ♦  fastest 

•  by  dropping  the  -ly  and  adding  -est: 

quickly  — ►  quickest 

•  by  using  the  word  most  before  the  adverb: 
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beautifully  — ►  most  beautifully 

Pattern:  List  [NP]  [Rel  Pronoun]  [VP]  [Adverb  +super]. 

Pattern:  [Wh-Word]  ([NP])  [VP]  [Adverb  +super]? 

Example  Who  was  hired  most  recently  ? 

Pattern:  [NP]  [VP]  [Adverb  -f  super]. 

Example  1:  The  Government  Officials  in  Honduras  are  kidnapped  most  often. 

Example  2:  (Preliminary  Sentence:  Guerrillas  attacked  several  farms.) 

The  Santo  Tomas  farm  was  burned  most  completely. 

Criteria:  Demonstrated  understanding  that  the  superlative  to  a  high  degree  expresses  an 
attribute  at  the  “high”  endpoint  of  the  scale  associated  with  that  attribute. 

_  Srore:  _ 


4.2  Superlative  Adverbs  to  a  Low  Degree 

A  superlative  to  a  low  degree  indicates  that  the  event  or  entity  to  whom  the  superlative 
applies  has  the  least  amount  of  an  attribute  or  comes  the  closest  to  the  “low”  endpoint  on 
the  scale  associated  with  an  attribute  about  which  the  comparison  is  made  (e.g.,  “least  easily 
sold”). 

Superlatives  to  a  low  degree  are  formed: 

•  with  the  word  least  before  the  adverb: 
often  — ►  least  often 

Pattern:  List  [NP]  [Rel  Pronoun]  [VP]  least  [Adverb]. 

Pattern:  [Wh-Word]  ([NP])  [VP]  least  [Adverb]? 

Example  Who  was  hired  least  recently  ? 

Pattern:  [NP]  [VP]  least 

Example  1:  The  Government  Officials  in  Honduras  are  kidnapped  least 

Example  2:  (Preliminary  Sentence:  Guerrillas  attacked  several  farms.) 

The  Santo  Tomas  farm  was  burned  least 

Criteria:  Demonstrated  understanding  that  the  superlative  to  a  low  degree  expresses  an 
attribute  at  the  “low”  endpoint  of  the  scale  associated  with  that  attribute. 

_ Score: _ 


[Adverb]. 

often. 

completely. 
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5.  THAN  as  a  Connective  Between  Mixed  Clauses 

In  this  section,  test  sentences  with  “than”  as  a  connective  between  a  copular  clause  ([NP] 
[BE- Verb]  [Adj])  and  a  non-copular  clause  ([NP]  [VP]).  The  comparison  you  use  may  be  to 
a  higher,  lower  or  equal  degree.  The  examples  given  here  are  to  a  higher  degree. 

Pattern:  List  [NP]i  [Rel  Pronoun]  [BE- Verb]  [Adj  +compare]i 

Example:  List  an  employee  who  is  more  experienced 

than  [NP]2  [VP], 

than  his  job  description  requires. 

Pattern:  [WH-Word]  ([NPji)  [BE- Verb]  [Adj  +compare]!  than  [NP]2  [VP]? 

Pattern:  [NP]  [Relative  Pronoun]  [BE- Verb]  [Adj  -|-compare]i  than  [NP]2  [VP]. 

Pattern:  [NP]  [Verb]  [NP]  [Relative  Pronoun]  [BE- Verb] 

Example:  Terrorists  killed  the  Government  official  who  is 

[Adj  +compare]|  than  [NP]2  [VP], 

more  Marxist  than  they  desired. 

Criteria:  Demonstrated  understanding  of  the  contrast  between  the  attribute  ([Adj  -f  com. 
pare]))  for  the  entity  expressed  by  the  noun  phrase  in  the  copular  clause  ([NP]  [BE- Verb] 
[Adj]),  and  the  attribute  expressed  by  the  non-copular  clause  ([NP]  [VP]). 

- - - - - Score:  _ 


6.  Comparative  Noun  Phrases  in  Object  Position 
The  standard  comparative  noun  phrase  pattern  is 

[NP]j  [Verb]i  [Object]i  than  [NP]2  [Verb]2  [Object]2 

where  [Objectji  and  [Object]2  are  noun  phrases  which  function  as  the  objects  of  [Verb]!  and 
[Verb]2,  respectively.  The  comparative  element  will  either  be  a  comparative  quantifier  within 
the  noun  phrase  functioning  as  [Object]i  (e.g.,  “more  printers ")  or  will  be  a  comparative 
adjective  as  a  premodifier  within  the  noun  phrase  functioning  as  [Object]i  (e.g.,  “older 
employee”). 

In  the  standard  comparative  noun  phrase  pattern,  u[NP]i  [Verb]!  [Object]i”  forms  a  clause 
and  “[NP]2  [Verb]2  [Object]2”  forms  another  clause.  In  this  section,  we  will  contrast  the 
noun  phrases  which  function  as  the  objects  of  the  two  clauses.  In  order  to  keep  the  tests 
as  simple  as  possible,  use  simple  nouns  phrases  for  the  [Object],  Remember  that  when  a 
grammatical  element  is  omitted,  it  is  understood. 
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6.1  Comparative  Noun  Phrases  Using  Comparative  Quantifiers 

Comparative  noun  phrases  using  comparative  quantifiers  use  the  grammatical  structures: 

Higher  quantity:  more  [Object]!  than  . . .  [Object]2 
Lower  quantity:  less  [Objectji  than  . . .  [Object]2 
Equal  quantity:  as  much  [Objectji  as  .. .  [Object]2 

The  following  four  subsections  test  comparative  noun  phrases  using  comparative  quantifiers 
without  gapping,  gapping  the  [Object]2,  gapping  the  [Verb]2  and  the  [Object]2,  and  gapping 
the  [NP]2  and  the  [Verb]2. 

COMPARATIVE  QUANTIFIER  NOUN  PHRASE  PATTERN: 

Pattern:  List  [NP]i  [Rel  Pronoun]  [Verb]!  more/less/as  much  [Objectji 

Example:  List  the  salesperson  who  sold  more  printers 

than/as  [NP]2  [Verb]2  [Object]2. 
than  Smith  sold  CPUs. 

Pattern:  [WH-Word]  ([NP]i)  [Verbji  more/less/as  much  [Objectji  than/as  [NP]2 
[Verb]2  [Object]2  ? 

[NP]i  [Verbji  more/less/as  much  [Objectji  than/as 

Shining  Path  killed  more  civilians  than 

(Preliminary  Sentence:  URNG  Guerrillas  attacked  six  farms.) 

The  guerrillas  attacked  more  farms  than 

[NP]2  [Verb]2  [Object]2. 

ARENA  kidnapped  Union  leaders, 

the  rebels  bombed  banks. 

6.1.1  No  Gapping 

Test  comparative  noun  phrases  which  have  comparative  quantifiers  with  no  gapping,  using 
the  COMPARATIVE  QUANTIFIER  NOUN  PHRASE  PATTERN. 


Pattern: 
Example  1: 
Example  2: 


Eg,  List  the  salesperson  who  sold  more  printers  than  Smith 
sold  CPUs. 

Eg,  Shining  Path  killed  more  civilians  than  ARENA  kidnapped  Union 
leaders . 

Eg,  (Preliminary  Sentence:  URNG  guerrillas  attacked  six  farms.) 

The  guerrillas  attacked  more  farms  than  the  rebels  bombed  banks. 


Criteria:  Demonstrated  understanding  within  the  context  of  the  sentence  of  the  contrast 
between  the  quantities  of  the  objects  of  the  two  clauses. 
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Score: 


6.1.2  Object  Gapping 

Test  comparative  noun  phrases  which  have  comparative  quantifiers  with  object  gapping  by 
omitting  the  [Object]2  from  the  COMPARATIVE  QUANTIFIER  NOUN  PHRASE  PAT¬ 
TERN. 

Eg.  List  the  salesperson  who  sold  more  printers  than  Smith 
sold. 

Eg,  Shining  Path  killed  more  civilians  than  ARENA  kidnapped. 

Eg,  (Preliminary  Sentence:  URNG  guerrillas  attacked  six  farms.) 

The  guerrillas  attacked  more  farms  than  the  rebels  attacked. 

Criteria:  Demonstrated  understanding  within  the  context  of  the  sentence  of  the  contrast 
between  the  quantity  of  the  object  of  the  first  clause,  and  the  quantity  of  the  omitted  (but 
implied)  object  of  the  second  clause. 

_ _ Score: _ 


6.1.3  Verb  and  Object  Gapping 

Test  comparative  noun  phrases  which  have  comparative  quantifiers  with  verb  and  object 
gapping  by  omitting  the  [Verb]2  and  the  [Object]2  from  the  COMPARATIVE  QUANTIFIER 
NOUN  PHRASE  PATTERN. 


Eg.  List  the  salesperson  who  sold  more  printers  them  Smith. 

Eg,  Shining  Path  killed  more  civilians  than  ARENA. 

Eg,  (Preliminary  Sentence:  URNG  guerrillas  attacked  six  farms.) 

The  guerrillas  attacked  more  farms  than  the  rebels. 

Criteria:  Demonstrated  understanding  within  the  context  of  the  sentence  of  the  contrast 
between  the  quantity  of  the  object  of  the  first  clause,  and  the  quantity  of  the  omitted  (but 
implied)  object  of  the  second  clause,  with  respect  to  the  action  expressed  by  the  verb. 

- — - — - Score: _ 


6.1.4  Noun  Phrase  and  Verb  Gapping 

Test  comparative  noun  phrases  which  have  comparative  quantifiers  with  noun  phrase  and 
verb  gapping  by  omitting  the  [NP]2  and  the  [Verb]2  from  the  COMPARATIVE  QUANTI¬ 
FIER  NOUN  PHRASE  PATTERN. 
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Eg,  List  the  salesperson  who  sold  more  printers  than  CPUs. 

Eg,  Shining  Path  killed  more  civilians  than  union  leaders. 

Eg,  (Preliminary  Sentence:  URNG  guerrillas  attacked  six  farms.) 

The  guerrillas  attacked  more  farms  than  banks. 

Criteria:  Demonstrated  understanding  within  the  context  of  the  sentence  of  the  contrast 
between  the  quantity  of  the  objects  of  the  two  clauses,  with  respect  to  the  action  expressed 
by  the  verb. 

_ _  -  Score:  _ 


6.2  Comparative  Noun  Phrases  Using  Comparative  Adjectives 

[Adj  “f" compare]  represents  a  comparative  adjective  to  a  higher,  lower,  or  same  degree. 
The  following  four  subsections  test  comparative  noun  phrases  using  comparative  adjectives 
without  gapping,  gapping  [Object]2,  gapping  [Verb]2  and  [Object]2,  and  gapping  [NP]2  and 
[Verb]2,  respectively. 

Use  the  -cr  form  of  the  comparative,  to  avoid  confusion  between  the  more  form  of  the 
comparative  and  more  used  as  a  quantifier. 

COMPARATIVE  ADJECTIVE  NOUN  PHRASE  PATTERN: 

Pattern:  List  [NP]i  [Rel  Pronoun]  [Verbjx  (Adj  -^compare]  [Objectjx  than 
[NP]2  [Verb]2  [Object]2. 

Pattern:  [WH-Word]  ([NP]i)  [Verbjx  [Adj  +compare]  [Object]i 

Example:  Which  department  pays  a  higher  commission 

than  [NP]2  [Verb]2  [Object]2  ? 
than  Smith  receives  a  salary  ? 

Pattern:  [NPjx  [Verbji  [Adj  Icompare]  [Objectjx  than 

Example  1:  The  FMLN  killed  a  higher-ranking  government  official  than 

Example  2:  (Preliminary  Sentence:  A  rebel  bombing  killed  two  civilians.) 

The  rebels  exploded  a  deadlier  grenade  than 

[NP]2  [Verb]2  [Object]2. 

the  ELN  killed  a  union  leader, 

guerrillas  exploded  a  bomb. 

6.2.1  No  Gapping 

Test  comparative  noun  phrases  which  have  comparative  adjectives  with  no  gapping,  using 
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the  COMPARATIVE  ADJECTIVE  NOUN  PHRASE  PATTERN. 


Eg,  Which  department  pays  a  higher  commission  than  Smith  receives 
a  salary? 

Eg,  The  FMLN  killed  a  higher-ranking  government  official  than  ELN 
killed  a  Union  leader. 

Eg,  (Preliminary  Sentence:  A  rebel  bombing  killed  tvo  civilians.) 
The  rebels  exploded  a  deadlier  grenade  than  the  guerrillas 
exploded  a  bomb. 


Criteria:  Demonstrated  understanding  within  the  context  of  the  sentence  of  the  contrast 
between  the  objects  of  the  two  clauses  with  respect  to  the  attribute  expressed  by  the  com¬ 
parative  adjectives. 

_ Score: _ 


6.2.2  Object  Gapping 

Test  comparative  noun  phrases  which  have  comparative  adjectives  with  object  gapping  by 
omitting  the  [Object]2  from  the  COMPARATIVE  ADJECTIVE  NOUN  PHRASE  PAT¬ 
TERN. 

Eg,  Which  department  pays  a  higher  commission  than  Smith  receives? 
Eg,  The  FMLN  killed  a  higher-ranking  government  official  than  ELN 
killed. 

Eg,  (Preliminary  Sentence:  A  rebel  bombing  killed  two  civilians.) 
The  rebels  exploded  a  deadlier  grenade  than  the  guerrillas 
exploded . 

Criteria:  Demonstrated  understanding  within  the  context  of  the  sentence  of  the  contrast 
between  the  object  of  the  first  clause  and  the  omitted  (but  implied)  object  of  the  second 
clause  with  respect  to  the  attribute  expressed  by  the  comparative  adjectives. 

_ Score:  _ 


6.2.3  Verb  and  Object  Gapping 

Test  comparative  noun  phrases  which  have  comparative  adjectives  with  verb  and  object 
gapping  by  omitting  the  [Verb]2  and  the  [Object)2  from  the  COMPARATIVE  ADJECTIVE 
NOUN  PHRASE  PATTERN. 


Eg,  Which  department  pays  a  higher  commission  than  the  New  York 
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branch? 

Eg,  The  FMLN  killed  a  higher-ranking  government  official  than  ELN. 
Eg,  (Preliminary  Sentence:  A  rebel  bombing  killed  tvo  civilians.) 
The  rebels  exploded  a  deadlier  grenade  than  the  guerrillas. 

Criteria:  Demonstrated  understanding  within  the  context  of  the  sentence  of  the  contrast 
between  the  object  of  the  first  clause  and  the  omitted  (but  implied)  object  of  the  second 
clause  with  respect  to  the  attribute  expressed  by  the  comparative  adjectives. 

_ Score: _ 


6.2.4  Noun  Phrase  and  Verb  Gapping 

Test  comparative  noun  phrases  which  have  comparative  adjectives  with  noun  phrase  and 
verb  gapping  by  omitting  the  [NP]2  and  the  [Verb]2  from  the  COMPARATIVE  ADJECTIVE 
NOUN  PHRASE  PATTERN. 

Eg,  Which  department  pays  a  higher  commission  than  a  salary? 

Eg.  The  FMLN  killed  a  higher-ranking  government  official  than 
a  union  leader. 

Eg,  (Preliminary  Sentence:  A  rebel  bombing  killed  tvo  civilians.) 
The  rebels  exploded  a  deadlier  grenade  than  the  bomb. 

Criteria:  Demonstrated  understanding  within  the  context  of  the  sentence  of  the  contrast 
between  the  objects  of  the  two  clauses  with  respect  to  the  attribute  expressed  by  the  com¬ 
parative  adjectives. 

- Score: _ 


7.  Comparing  Objects  of  VPs  with  Extra  Adjuncts 

In  this  section,  we  will  contrast  the  noun  phrases  which  function  as  the  objects  in  the  two 
comparative  clauses.  Thus,  section  7  is  similar  to  section  6,  with  the  exception  of  an  extra 
adjunct  appearing  in  both  clauses  of  the  comparative  sentence.  What  is  being  tested  is  the 
additional  adjunct.  It  is  therefore  unnecessary  to  test  both  the  comparative  quantifier  and 
the  comparative  adjective  versions  of  the  noun  phrase  in  object  position  test.  For  testing 
purposes,  use  sentences  from  either  the  MAIN  COMPARATIVE  QUANTIFIER  PATTERNS 
or  the  MAIN  COMPARATIVE  ADJECTIVE  PATTERNS. 

MAIN  COMPARATIVE  QUANTIFIER  PATTERNS: 

Pattern:  List  [NP]i  [Rel  Pronoun]  [Verb]j  more  [Object]i  [Adjunct^ 

Example:  List  the  salesperson  who  sold  more  printers  in  1990 
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than  [NP]2  [Verb]2  [Object]2  [Adjunct]2. 
than  Smith  sold  CPUs  in  1989. 


Pattern:  [WH-Word]  ([NP]j)  [Verb]]  more  [Object]i  [Adjunct]]  than 

Example:  Which  salesperson  sold  more  printers  in  1989  than 

[NP]2  [Verb]2  [Object]2  [Adjunct]2? 

John  Smith  sold  CPUs  in  1990? 


Pattern: 
Example  1: 
Example  2: 


[NP]i  [Verb]x  more  [Object]]  [Adjunct]]  than 

Shining  Path  killed  more  civilians  in  Honduras  than 

(Preliminary  Sentence:  URNG  Guerrillas  attacked  six  farms.) 

The  guerrillas  attacked  more  buildings  on  the  presidential  farm  than 

[NP]2  [Verb]2  [Object]2  [Adjunct]2. 

ARENA  kidnapped  union  leaders  in  El  Salvador. 

the  rebels  bombed  buildings  on  the  Santo  Tomas  farm. 


MAIN  COMPARATIVE  ADJECTIVE  PATTERNS: 

Pattern:  List  [NP]X  [Rel  Pronoun]  [Verb]]  [Adj  ^compare]  [Object]]  [Adjunct]] 

Example:  List  the  branch  which  paid  a  higher  commission  in  1989 

than  [NP]2  [Verb]2  [Object]2  [Adjunct]2. 
than  Jones  received  salary  in  1990. 


Pattern:  [WH-Word]  [NP]X  [Verb]]  [Adj  +compare]  [Object]]  [Adjunct]] 

Example:  Which  department  paid  a  higher  commission  in  February 

than  [NP]2  [Verb]2  [Object]2  [Adjunct]2  ? 

than  Jones  received  a  salary  in  March  ? 

Pattern:  [NP]j  [Verb]]  [Adj  +compare]  [Object]]  [Adjunct]]  than 

Example  1:  The  FMLN  killed  a  higher-ranking  official  in  December  than 

Example  2:  (Preliminary  Sentence:  A  rebel  bombing  killed  two  civilians.) 

The  rebels  exploded  a  deadlier  grenade  on  January  2  than 

[NP]2  [Verb]2  [Object]2  [Adjunct]2. 

the  ELN  killed  a  union  leader  in  November, 
guerrillas  exploded  a  bomb  on  February  10. 


7.1  No  Gapping 

Test  comparative  noun  phrases  with  an  extra  adjunct  with  no  gapping.  Use  either  the  MAIN 
COMPARATIVE  QUANTIFIER  PATTERNS  or  the  MAIN  COMPARATIVE  ADJECTIVE 
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PATTERNS. 


Eg.  List  the  salesperson  vho  sold  more  printers  in  1990  than  Smith 
sold  CPUs  in  1989. 

Eg,  Which  salesperson  sold  more  printers  in  1989  than  John  Smith 
sold  CPUs  in  1990. 

Eg,  Shining  Path  killed  more  civilians  in  Honduras  than  ARENA 
kidnapped  union  leaders  in  El  Salvador. 

Eg.  List  the  department  vhich  paid  a  higher  commission  in  1990  than 
Jones  received  a  salary  in  1989. 

Eg,  Which  department  paid  a  higher  commission  in  February  than 
Jones  received  a  salary  in  March. 

Eg,  The  FMLN  killed  a  higher-ranking  government  official  in  December 
than  ELN  killed  a  union  leader  in  November. 

Eg,  (Preliminary  Sentence:  URNG  Guerrillas  attacked  six  farms.)  The 
guerrillas  attacked  more  buildings  on  the  presidential  farm  than 
the  rebels  bombed  buildings  on  the  Santo  Tomas  farm. 

Eg,  (Preliminary  Sentence:  A  rebel  bombing  killed  two  civilians.) 

The  rebels  exploded  a  deadlier  grenade  on  January  2  than 
guerrillas  exploded  a  bomb  on  February  10. 

Criteria:  Demonstrated  understanding  within  the  context  of  the  sentence  of  either 

•  the  contrast  between  the  quantities  of  the  objects  of  the  two  clauses. 

or 

•  the  contrast  between  the  objects  of  the  two  clauses  with  respect  to  the  attribute 

expressed  by  the  comparative  adjectives. 

_ Score: _ 


7.2  Single  Gapping 

In  these  sentences  with  a  subject,  verb,  object,  and  adjunct,  there  are  four  possible  single 
gappings,  each  one  formed  by  omitting  one  of  these  four  sentence  parts.  The  subject  and 
the  object  are  noun  phrases.  Using  one  of  the  above  pattern  sets,  test  for  the  four  possible 
single  gappings  by  omitting  each  of  the  four  phrases  in  the  than  clause  in  turn,  generating 
the  following  alternatives,  where  “0”  represents  an  omitted  item: 

than  0  [Verb]  [Object]  [Adjunct] 

than  [Subject]  0  [Object]  [Adjunct] 

than  [Subject]  [Verb]  0  [Adjunct] 

than  [Subject]  [Verb]  [Object]  0 

For  example,  the  following  sentences  illustrate  the  first  case,  above,  where  the  subject  is 
omitted  from  the  second  clause: 

Eg.  List  the  salesperson  who  sold  more  printers  in  1990 
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than  sold  CPUs  in  1989. 

Eg,  Shining  Path  killed  more  civilians  in  Honduras  than  kidnapped 
union  leaders  in  El  Salvador. 

Eg,  Which  department  paid  a  higher  commission  in  February  than 
paid  a  salary  in  March. 

Eg,  The  FMLN  killed  a  higher-ranking  government  official  in  December 
than  killed  a  union  leader  in  November. 

Eg,  (Preliminary  Sentence:  URNG  Guerrillas  attacked  six  farms.)  The 
guerrillas  attacked  more  buildings  on  the  presidential  f arm  than 
bombed  buildings  on  the  Santo  Tomas  farm. 

Eg,  (Preliminary  Sentence:  A  rebel  bombing  killed  two  civilians.) 

The  rebels  exploded  a  deadlier  grenade  on  January  2  than 
exploded  a  bomb  on  February  10. 

Criteria:  Demonstrated  understanding  within  the  context  of  the  sentence  of  either 

•  the  contrast  between  the  quantities  of  the  objects  of  the  two  clauses. 

or 

•  the  contrast  between  the  objects  of  the  two  clauses  with  respect  to  the  attribute 

expressed  by  the  comparative  adjectives. 

. — - - - - — — _ Score: _ 


Criteria:  Demonstrated  understanding  within  the  context  of  the  sentence  of  either 

•  the  contrast  between  the  quantities  of  the  objects  of  the  two  clauses. 

or 

•  the  contrast  between  the  objects  of  the  two  clauses  with  respect  to  the  attribute 

expressed  by  the  comparative  adjectives. 

- - - - - - Score: _ 


Criteria:  Demonstrated  understanding  within  the  context  of  the  sentence  of  either 

•  the  contrast  between  the  quantity  of  the  object  of  the  first  clause  and  the  quantity 

of  the  omitted  (but  implied)  object  of  the  second  clause. 

or 

•  the  contrast  between  the  object  of  the  first  clause  and  the  omitted  (but  implied) 

object  of  the  second  clause  with  respect  to  the  attribute  expressed  by  the  compar¬ 
ative  adjectives. 

- Score: _ 


Criteria:  Demonstrated  understanding  within  the  context  of  the  sentence  of  either 
•  the  contrast  between  the  quantities  of  the  objects  of  the  two  clauses. 
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or 

•  the  contrast  between  the  objects  of  the  two  clauses  with  respect  to  the  attribute 
expressed  by  the  comparative  adjectives. 

_  Score:  _ 


7.3  Double  Gapping 

In  these  sentences  with  a  subject,  verb,  object,  and  adjunct,  there  are  six  possible  double 
gappings,  each  one  formed  by  omitting  two  of  these  four  sentence  parts.  Using  one  of  the 
above  pattern  sets,  test  for  the  six  possible  double  gappings  by  omitting  a  pair  of  the  four 
phrases  in  the  than  clause  in  turn,  generating  these  alternatives: 


than 

0 

0 

[Object] 

[Adjunct] 

than 

0 

[Verb] 

0 

[Adjunct] 

than 

0 

[Verb] 

[Object] 

0 

than 

[Subject] 

0 

0 

[Adjunct] 

than 

[Subject] 

0 

[Object] 

0 

than 

[Subject] 

[Verb] 

0 

0 

For  example,  the  following  sentences  illustrate  the  first  case,  above,  where  the  subject  and 
verb  are  omitted  from  the  second  clause: 

Eg.  List  the  salesperson  vho  sold  more  printers  in  1990 
than  CPUs  in  1989. 

Eg,  Shining  Path  killed  more  civilians  in  Honduras  than  union 
leaders  in  El  Salvador. 

Eg,  Which  department  paid  a  higher  commission  in  February  than 
salary  in  March. 

Eg,  The  FMLN  killed  a  higher-ranking  government  official  in  December 
a  union  leader  in  November. 

Eg,  (Preliminary  Sentence:  URNG  Guerrillas  attacked  six  farms.)  The 
guerrillas  attacked  more  buildings  on  the  presidential  farm  than 
buildings  on  the  Santo  Tomas  farm. 

Eg,  (Preliminary  Sentence:  A  rebel  bombing  killed  two  civilians.) 

The  rebels  exploded  a  deadlier  grenade  on  January  2  than 
a  bomb  on  February  10. 

Criteria:  Demonstrated  understanding  within  the  context  of  the  sentence  of  either 

•  the  contrast  between  the  quantities  of  the  objects  of  the  two  clauses. 

or 

•  the  contrast  between  the  objects  of  the  two  clauses  with  respect  to  the  attribute 

expressed  by  the  comparative  adjectives. 

- — - Score: _ 
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Criteria:  Demonstrated  understanding  within  the  context  of  the  sentence  of  either 

•  the  contrast  between  the  quantity  of  the  object  of  the  first  clause  and  the  quantity 

of  the  omitted  (but  implied)  object  of  the  second  clause. 

or 

•  the  contrast  between  the  object  of  the  first  clause  and  the  omitted  (but  implied) 

object  of  the  second  clause  with  respect  to  the  attribute  expressed  by  the  compar¬ 
ative  adjectives. 

_ Score: _ 


Criteria:  Demonstrated  understanding  within  the  context  of  the  sentence  of  either 

•  the  contrast  between  the  quantities  of  the  objects  of  the  two  clauses. 

or 

•  the  contrast  between  the  objects  of  the  two  clauses  with  respect  to  the  attribute 

expressed  by  the  comparative  adjectives. 

_ Score: _ 


Criteria:  Demonstrated  understanding  within  the  context  of  the  sentence  of  either 

•  the  contrast  between  the  quantity  of  the  object  of  the  first  clause  and  the  quantity 

of  the  omitted  (but  implied)  object  of  the  second  clause. 

or 

•  the  contrast  between  the  object  of  the  first  clause  and  the  omitted  (but  implied) 

object  of  the  second  clause  with  respect  to  the  attribute  expressed  by  the  compar¬ 
ative  adjectives. 

_ Score: _ 


Criteria:  Demonstrated  understanding  within  the  context  of  the  sentence  of  either 

•  the  contrast  between  the  quantities  of  the  objects  of  the  two  clauses. 

or 

•  the  contrast  between  the  objects  of  the  two  clauses  with  respect  to  the  attribute 

expressed  by  the  comparative  adjectives. 

_ Score:  _ 


Criteria:  Demonstrated  understanding  within  the  context  of  the  sentence  of  either 

•  the  contrast  between  the  quantity  of  the  object  of  the  first  clause  and  the  quantity 
of  the  omitted  (but  implied)  object  of  the  second  clause. 
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or 

•  the  contrast  between  the  object  of  the  first  clause  and  the  omitted  (but  implied) 
object  of  the  second  clause  with  respect  to  the  attribute  expressed  by  the  compar¬ 
ative  adjectives. 

_ Score: _ 


7.4  Triple  Gapping 

In  these  sentences  with  a  subject,  verb,  object,  and  adjunct,  there  are  four  possible  triple 
gappings,  each  formed  by  omitting  three  of  these  four  sentence  parts.  Using  one  of  the  above 
pattern  sets,  test  for  the  four  possible  triple  gappings  by  omitting  three  of  the  four  phrases 
in  the  than  clause  in  turn,  generating  these  alternatives: 

than  [Subject]  0  0  0 

than  0  [Verb]  0  0 

than  0  0  [Object]  0 

than  0  0  0  [Adjunct] 

For  example,  the  following  sentences  illustrate  the  first  case,  above,  where  the  verb,  the 
object,  and  the  adjunct  are  omitted  from  the  second  clause: 

Eg,  List  the  salesperson  who  sold  more  printers  in  1990 
than  Smith. 

Eg,  Shining  Path  killed  more  civilians  in  Honduras  than  ARENA. 

Eg,  Which  department  paid  a  higher  commission  in  February  them 
department  12. 

Eg,  The  FMLN  killed  a  higher-ranking  government  official  in  December 
than  ELN. 

Eg,  (Preliminary  Sentence:  URNG  Guerrillas  attacked  six  farms.)  The 
guerrillas  attacked  more  buildings  on  the  presidential  farm  than 
workers . 

Eg,  (Preliminary  Sentence:  A  rebel  bombing  killed  two  civilians.) 

The  rebels  exploded  a  deadlier  grenade  on  January  2  than  a  bomb. 

Criteria:  Demonstrated  understanding  within  the  context  of  the  sentence  of  either 

•  the  contrast  between  the  quantity  of  the  object  of  the  first  clause  and  the  quantity 

of  the  omitted  (but  implied)  object  of  the  second  clause. 

or 

•  the  contrast  between  the  object  of  the  first  clause  and  the  omitted  (but  implied) 

object  of  the  second  clause  with  respect  to  the  attribute  expressed  by  the  compar¬ 
ative  adjectives. 

— — — - Score: _ 
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Criteria:  Demonstrated  understanding  within  the  context  of  the  sentence  of  either 

•  the  contrast  between  the  quantity  of  the  object  of  the  first  clause  and  the  quantity 

of  the  omitted  (but  implied)  object  of  the  second  clause, 
or 

•  the  contrast  between  the  object  of  the  first  clause  and  the  omitted  (but  implied) 

object  of  the  second  clause  with  respect  to  the  attribute  expressed  by  the  compar¬ 
ative  adjectives. 

_ _ _ _  Score: _ 


Criteria:  Demonstrated  understanding  within  the  context  of  the  sentence  of  either 

•  the  contrast  between  the  quantities  of  the  objects  of  the  two  clauses. 

or 

•  the  contrast  between  the  objects  of  the  two  clauses  with  respect  to  the  attribute 

expressed  by  the  comparative  adjectives. 

_  _ _ Score: _ 


Criteria:  Demonstrated  understanding  within  the  context  of  the  sentence  of  either 

•  the  contrast  between  the  quantity  of  the  object  of  the  first  clause  and  the  quantity 

of  the  omitted  (but  implied)  object  of  the  second  clause. 

or 

•  the  contrast  between  the  object  of  the  first  clause  and  the  omitted  (but  implied) 

object  of  the  second  clause  with  respect  to  the  attribute  expressed  by  the  compar¬ 
ative  adjectives. 

_ Score: _ 


8.  Implicit  Comparatives 

Implicit  comparatives  are  certain  adjectives  such  as  the  words  senior,  junior,  inferior,  supe¬ 
rior,  prior,  anterior,  and  posterior. 

Test  an  implicit  comparative.  You  may  use  any  sentence  pattern  which  has  been  successfully 
tested.  Below  are  some  suggested  sentence  patterns. 

Pattern:  List  [NP]i  [Rel  Pronoun]  [BE- Verb]  [Implicit  Comparative] 

Example:  List  an  employee  who  is  senior 

to  [NP]2. 
to  John  Doe. 
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Pattern:  [WH-Word]  ([NP]i)  [BE- Verb]  [Implicit  Comparative]  to  [NP]2  ? 


Pattern:  [NP]  [Verb]  [NP]i  ([Relative  Pronoun]) 

Example  1:  The  terrorists  assassinated  the  government  official  who 

Example  2:  URNG  guerrillas  conducted  an  attack 

([BE- Verb])  [Implicit  Comparative]  to  [NP]2. 

is  superior  to  the  Attorney  General. 

prior  to  the  February  2  attack. 

Criteria:  Demonstrated  understanding  of  the  contrast  between  the  two  entities  expressed  by 
[NP]i  and  [NP]2  with  respect  to  the  attribute  expressed  by  the  implicit  comparative. 

_ Score: _ 


9.  Multipliers  and  Fractions  in  Comparisons 

In  the  following  subsections,  we  compare  two  actions,  entities,  or  attributes  using  a  multiplier 
or  a  fraction  to  further  define  a  numerical  measure. 

Choose  from  the  following  types  of  comparative  elements,  denoted  [Comparative  Element]: 

•  The  quantifier  “many”, 

•  The  quantifier  “many”  followed  by  a  noun  phrase,  as  in: 

many  employees 

•  An  adjective. 

9.1  Multipliers 

Test  a  comparative  which  uses  a  multiplier.  You  may  use  any  sentence  pattern  which  has 
been  successfully  tested.  Below  are  some  suggested  sentence  patterns. 

Pattern:  List  [NP]j  [Rel  Pronouns]  [Verb]  [Multiplier]  as 

Example:  List  the  employees  that  are  twice  as 

[Comparative  Element]  as  [NP]2  ([VP]), 

old  as  John  Doe. 

Pattern:  [WH-Word]  ([NP]i)  [Verb]  [Multiplier]  as  [Comparative  Element] 

Example:  Who  earns  twice  as  much  salary 

as  [NP]2  ([VP])  ? 

as  John  Smith  ? 

Pattern:  [NP]  [Verb]  [Multiplier]  as  [Comparative  Element] 

Example:  Shining  Path  killed  three  times  as  many  civilians 


as 


[NP],  ([VP]), 

as  ARENA. 

Criteria:  Demonstrated  understanding  of  the  numerical  measure  expressed  by  the  multipli¬ 
cation  applied  to  the  comparative  element. 

_ Score: _ 

9.2  Fractions 

Test  a  comparative  which  uses  a  fraction.  You  may  use  any  sentence  pattern  which  has  been 
successfully  tested.  Below  are  some  suggested  sentence  patterns. 

Pattern:  List  [NP]i  [Rel  Pronouns]  [Verb]  [Fraction]  as 

Example:  List  the  employees  that  are  two-thirds  as 

[Comparative  Element]  as  [NP]2  ([VP]), 

old  as  John  Doe. 

Pattern:  [WH-Word]  ([NP]!)  [Verb]  [Fraction]  as  [Comparative  Element] 

Example:  Who  earns  half  as  much  salary 

as  [NP]2  ([VP])  ? 

as  John  Smith  ? 

Pattern:  [NP]  [Verb]  [Fraction]  as  [Comparative  Element] 

Example:  Shining  Path  killed  one  third  as  many  civilians 

as  [NP],  ([VP]), 

as  ARENA. 

Criteria:  Demonstrated  understanding  of  the  numerical  measure  expressed  by  the  fraction 
applied  to  the  comparative  element. 

_  Score:  _ 


10.  Comparisons  Using  TOO  (t.b.d.) 
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VIII.  CONNECTIVES 
1  Coordinators  (Single  Pair  Coordination) 

By  “single  pair  coordination”  we  refer  to  coordination  with  just  two  elements.  Coordination 
with  greater  than  two  elements  is  covered  in  a  later  section. 

“Scope”  is  the  semantic  range  of  influence  which  certain  words  have  over  other  parts  of  a 
sentence  [Quirk  et  al  1985,  p.  85].  In  the  sentence: 

What  employees  hire  and  supervise  assistants ? 

the  object  “assistants”  is  contained  within  the  scope  of  the  verb  “ hire ”,  so  that  the  noun 
“ assistants ”  functions  as  the  object  of  the  verb  “hire”.  That  is,  assistants  are  are  hired  as 
well  as  supervised. 

Note  the  use  of  an  extra  square  bracket  pair,  “[  ]”,  in  many  of  the  test  patterns.  We  use 
brackets  to  aid  in  the  determination  of  which  sentence  elements  are  being  conjoined.  The 
elements  inside  the  brackets  are  to  be  treated  as  one  object.  Thus,  the  phrase: 

young  [men  and  women] 

has  the  interpretation  that  the  men  and  women  are  young.  The  nouns  “men”  and  “women” 
are  both  within  the  scope  of  the  adjective  “young”.  In  the  phrase: 

[young  men]  and  women 

the  brackets  show  that  the  noun  “men”  is  within  the  scope  of  the  adjective  “young”  while 
the  noun  “women”  is  not.  The  interpretation  is  that  the  men  are  young  and  the  women’s 
ages  are  unspecified. 

1.1  AND 

1.1.1  Sentential  Connectives 

1.1. 1.1  Declaratives  Conjoined  by  AND 

Test  two  declarative  sentences  connected  by  the  word  “and” .  Use  any  two  that  have  been 
processed  successfully.  Be  sure  that  the  two  sentences  are  full  sentences  without  any  omis¬ 
sions. 


Pattern: 

[Declarative]] 

and 

[Declarative^. 

Example: 

Jane  Smith  is  a  manager 

and 

John  Doe  is  a  salesperson. 

Example: 

Jane  Smith  earned  $21,000  in  1989 

and 

John  Doe  earned  $20,000  in  1989 

Example: 

Terrorists  attacked  the  Government 
Building 

and 

20  men  were  killed. 

Example: 

Guerrillas  attacked  a  farm 

and 

a  peasant  was  killed. 

Example: 

Guerrillas  bombed  the  Central  Bank 

and 

two  civilians  were  injured. 
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Criteria:  Demonstrated  understanding  of  the  declarative  sentences  conjoined  with  “and.”. 
_ Score: _ 


1.1. 1.2  Imperatives  Conjoined  by  AND 

Test  two  imperative  sentences  connected  by  the  word  “and”.  Use  any  two  that  have  been 
processed  successfully.  Be  sure  that  the  two  sentences  are  full  sentences  without  any  omis¬ 
sions. 

Pattern:  [Imperative^  and  [Imperative]2. 

Example:  List  the  men  in  Chicago  and  list  the  women  in  New  York. 

Example:  List  the  employees  and  tell  me  the  salaries. 

Criteria:  Demonstrated  understanding  of  the  imperative  sentences  conjoined  with  “and”. 

_  Score:  _ 


1.1. 1.3  Interrogative?  Conjoined  by  AND 

Test  two  interrogative  clauses  connected  by  the  word  “and” .  Use  any  two  that  have  already 
been  processed  successfully.  Make  sure  the  two  interrogatives  are  complete  sentences. 

Pattern:  [Interrogative] i  and  [Interrogative]2  ? 

Example:  Who  is  the  oldest  employee  and  what  do  the  Chicago  employees  earn? 
Example:  What  employees  are  women  and  who  works  in  the  New  York  branch? 

Criteria:  Demonstrated  understanding  of  the  interrogative  sentences  conjoined  with  “and”. 

_ _  Score: _ 


1.1.2  Relative  Clauses  Conjoined  by  AND 

Test  conjoined  relative  clauses.  Remember  that  relative  clauses  function  as  postmodifiers 
in  a  noun  phrase.  You  may  choose  from  the  following  sentence  patterns  or  choose  any 
successful  sentence  pattern  from  a  previous  section.  Note  in  the  examples  which  follow  that 
both  relative  clauses  modify  the  noun  phrase,  as  indicated  by  the  brackets  enclosing  the 
conjoined  relative  clauses. 

Remember  that  [Verb  (-ed)]  refers  to  the  passive  form  of  the  verb,  the  verb  with  the  -ed 
participle.  For  more  information  on  the  passive  verb ,  see  Section  I,  Subsection  4.4  Voice. 

Pattern:  List  [NP]  [[Relative  Clause]  and  [Relative  Clause]]. 

Example:  List  the  employees  who  earn  $25,000  and  who  work  in  New  York. 
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Pattern: 

Who 

[BE-Verb] 

[NP] 

[[Relative  Clause] 

and 

[Relative  Clause]]  ? 

Example: 

Who 

are 

the  employees 

who  earn  $25,000 

and 

who  work  in  New  York  ? 

Pattern: 

What 

[BE-Verb] 

[NP] 

[[Relative  Clause] 

and 

[Relative  Clause]] 

Example: 

What 

are 

the  employees 

who  are  women 

and 

who  are  educated 

[Verb  (-ed)]? 
paid? 


Pattern:  What  ([NP])  [DO- Verb]  [NP]  [[Relative  Clause]  and 

Example:  What  salaries  do  the  employees  who  live  in  New  York  and 

[Relative  Clause]]  [Verb]? 
who  have  a  degree  earn? 


Pattern:  [NP]  [[Relative  Clause]  and 

Example:  The  terrorists  who  attacked  the  Government  House  and 
Example:  The  guerrillas  who  burned  the  presidential  farm  and 

[Relative  Clause]]  [VP]. 

who  killed  20  people  bombed  the  Central  Bank. 

who  stole  food  killed  a  peasant. 


Pattern: 

Example: 

Example: 


[NP]  [Verb]  [NP]  [[Relative  Clause]  and 

Peasants  shot  the  men  who  bombed  the  Bank  and 

The  police  arrested  the  rebels  who  burned  vehicles  and 

[Relative  Clause]], 
who  killed  20  people, 
who  robbed  shops. 


Criteria:  Demonstrated  understanding  of  the  qualification  of  the  postmodified  noun  phrase 
by  the  conjoined  relative  clauses. 


Score: 


1.1.3  Noun  Phrase  Constructions  with  the  Conjunction  AND 

Choose  one  of  the  sentence  patterns  from  the  group  listed  below  which  has  been  tested 
successfully  for  use  in  the  following  subsections.  In  each  of  the  subsections,  a  noun  phrase 
construction  with  the  conjunction  “and”  is  specified.  You  will  replace  the  [NP  -(-conjunction] 
in  the  chosen  pattern  with  the  specified  noun  phrase. 
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Pattern: 

Example: 

Example: 

Pattern: 

Example: 

Pattern: 

Example: 


Pattern: 

Example: 


Pattern: 

Example: 

Example: 

Pattern: 

Example: 

Example: 


List  [NP  4-conjunction]. 

List  the  men  and  women  in  New  York. 

List  the  married  men  and  the  single  women. 

Who  [BE- Verb]  [NP  -^conjunction]? 

Who  are  the  men  in  NY  and  the  women  in  Chicago? 

What  ([NP])  [BE- Verb]  [NP  4-conjunction]  [Verb  (-ed)]? 

What  are  the  men  and  women  paid? 

in  the  New  York  branch 

What  ([NP])  [DO-Verb]  [NP  -^-conjunction]  [Verb]? 

What  salaries  do  the  oldest  employee  and  earn? 

the  youngest  employee 

[NP  4-conjunction]  [Verb]  [NP]. 

The  terrorists  and  guerrillas  in  Bolivia  attacked  the  Government  House. 
Guerrillas  and  rebels  in  Guatemala  City  attacked  the  Central  Bank. 

[NP]  [Verb]  [NP  -(-conjunction]. 

Terrorists  in  Bolivia  bombed  the  residences  and  brothels. 

Guerrillas  killed  women  and  children  at  the  presidential  farm. 


Brackets  are  used  in  the  specified  noun  phrases  in  the  following  subsections  to  aid  in  the 
determination  of  which  elements  are  being  conjoined.  Also,  brackets  are  used  in  the  examples 
to  identify  the  phrases  which  follow  the  specified  noun  phrase  pattern. 

1.1. 3.1  Conjunction  of  Simple  Noun  within  the  Scope  of  an  Article 

Replace  the  [NP  4-conjunction]  in  your  chosen  sentence  pattern  with  the  noun  phrase: 
[Article]  [[Noun]  and  [Noun]]  ([Postmodification]) 

Eg,  List  [the  men  and  women  in  the  New  York  branch] . 

Eg,  What  are  [the  men  and  women  in  the  New  York  branch]  paid? 

Eg,  Terrorists  in  Bolivia  bombed  [the  residences  and  brothels] . 

Eg,  [The  terrorists  and  guerrillas  in  Bolivia]  attacked  the 
Government  House. 

Eg,  Guerrillas  killed  [the  women  and  children  at  the  presidential 
farm] . 

Eg,  [The  guerrillas  and  rebels  in  Guatemala  City]  attacked  the 
Central  Bank. 


Criteria:  Demonstrated  understanding  of  the  noun  phrase  formed  with  the  conjunction  of 
two  nouns. 
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Score: 


1.1. 3. 2  Conjunction  of  Nouns  with  Adjective  Premodifiers 

Replace  the  [NP  +conjunction]  in  your  chosen  sentence  pattern  with  the  noun  phrase: 

[Article]  [[Adjective]  [Noun]  and  [Adjective]  [Noun]]  ([Postmodifier]) 

Eg,  Who  are  [the  married  men  and  single  women  in  the  New  York 
branch] . 

Eg,  Terrorists  in  Bolivia  bombed  [the  countryside  residences  and 
downtown  brothels] . 

Eg,  [The  Maoist  terrorists  and  Marxist  guerrillas  in  Bolivia] 
attacked  the  Government  House. 

Eg,  The  guerrillas  bombed  [the  presidential  farm  and  Government 
House.] 

Eg,  [The  armed  guerrillas  and  militant  rebels]  burned  the  shops. 


Criteria:  Demonstrated  understanding  of  the  noun  phrase  formed  with  the  conjunction  of 
two  phrases  consisting  of  “[Adjective]  [Noun]”. 

_  Score:  _ 


1.1. 3. 3  Conjunction  of  Nouns  with  Articles  and  Adjective  Premodifiers 

Replace  the  [NP  -f conjunction]  in  your  chosen  sentence  pattern  with  the  noun  phrase: 

[[Article]  [Adjective]  [Noun]  and  [Article]  [Adjective]  [Noun]]  ([Postmodifier]) 

Eg,  What  are  [the  married  men  and  the  single  women  in  New  York] 
paid? 

Eg,  Terrorists  in  Guatemala  bombed  [the  countryside  residences  and 
the  downtown  brothels] . 

Eg,  [The  Maoist  terrorists  and  the  Marxist  guerrillas  in  Escuintla] 
attacked  the  Government  House. 

Eg,  Guerrillas  attacked  [the  presidential  farm  and  the  Government 
House] . 

Eg,  [The  armed  guerrillas  and  the  militant  rebels]  burned  shops. 

Criteria:  Demonstrated  understanding  of  the  noun  phrase  formed  with  the  conjunction  of 
two  phrases  consisting  of  “[Article]  [Adjective]  [Noun]”. 
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Score: 


1.1. 3.4  Conjunction  of  Noun  Phrases  with  Premodifiers  and  Postmodifiers 

Replace  the  [NP  + conjunction]  in  your  chosen  sentence  pattern  with  the  noun  phrase: 

[[Article]  [Adjective]  [Noun]  [Postmodification] 

and  [Article]  [Adjective]  [Noun]  [Postmodification]]. 

Eg,  What  do  [the  oldest  employee  in  department  17  and  the  youngest 
worker  in  the  payroll  department]  earn? 

Eg,  Terrorists  bombed  [the  countryside  residences  in  Cuilapa  and  the 
downtown  brothels  in  Escuintla] . 

Eg,  [The  Maoist  terrorists  in  Escuintla  and  the  Marxist  guerrillas 
in  Cuilapa]  attacked  the  electric  facilities. 

Eg,  The  guerrillas  bombed  [the  presidential  farm  outside  the  capital 
city  and  the  Government  House  in  the  city] . 

Eg,  [The  armed  guerrillas  in  Guatemala  City  and  the  militant 
rebels  in  Flores]  burned  shops. 


Criteria:  Demonstrated  understanding  of  the  noun  phrase  formed  with  the  conjunction  of 
*wo  phrases  consisting  of  “[Article]  [Adjective]  [Noun]  [Postmodification]” . 

_ Score: _ 


1.1.4  The  Conjunction  AND  in  a  Premodification  of  a  Noun  Phrase 


This  section  tests  the  use  of  conjunctions  within  the  premodification  of  a  noun  phrase.  In 
the  following  subsections,  a  noun  phrase  with  a  conjunction  in  its  premodifier  is  specified. 
You  will  replace  the  [NP  +premod]  in  the  chosen  pattern  with  the  specified  noun  phrase. 


Pattern:  List  [NP  +premod]. 

Example:  List  the  young  and  single  men  in  the  New  York  branch. 
Example:  List  the  single  and  the  married  men  who  sell  printers. 


Pattern:  Who  [BE- Verb]  [NP  4-premod]  ? 

Example:  Who  are  the  young  and  single  men  in  the  New  York  branch  ? 

Example:  Who  are  the  educated  and  experienced  men  in  New  York? 


Pattern:  What  ([NP])  [BE- Verb]  [NP  Ipremod]  [Verb  (-ed)]  ? 

Example:  What  salaries  are  the  young  and  single  men  paid? 


Pattern:  [NP  4-premod]  [Verb]  [NP]. 

Example:  The  young  and  militant  rebels  attacked  the  presidential  farm. 

Example:  The  FMLN  and  ELN  terrorists  attacked  the  Government  House. 
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Pattern:  [NP]  [Verb]  [NP  +premod]. 

Example:  Terrorists  attacked  the  Prime  Minister’s  and  the 

Secretary  General’s  residences. 

Brackets  are  used  in  the  specified  noun  phrases  in  the  following  subsections  to  aid  in  the 
determination  of  which  elements  are  being  conjoined.  Also,  brackets  are  used  in  the  examples 
to  identify  the  phrases  which  follow  the  specified  noun  phrase  pattern. 

1.1. 4.1  Conjunctions  of  Adjectives 

Chose  two  adjectives  which  together  will  describe  one  object  or  group  expressed  by  the  noun 
which  they  premodify.  For  example,  in 

young  and  single  men 

both  “young”  and  “single”  describe  the  same  group  of  men. 

Replace  the  [NP  +premod]  in  your  chosen  sentence  pattern  with  the  noun  phrase: 

[Article]  [[Adjective]  and  [Adjective]]  [Noun]  ([Postmodification]) 

Eg,  List  [the  young  and  single  men  in  the  New  York  branch] . 

Eg,  Who  are  [the  educated  and  experienced  women]? 

Eg,  [The  young  and  militant  rebels]  attacked  the  presidential  farm. 

Criteria:  Demonstrated  understanding  of  the  qualification  of  the  premodified  noun  phrase 
by  the  conjoined  adjectives. 

_ Score:  _ 


1.1. 4. 2  Conjunctions  of  Article  with  Adjective 

Chose  two  adjectives  which  define  different  objects  or  groups  expressed  by  the  noun  which 
they  premodify.  For  example,  in 

single  and  married  men 

“single”  describes  one  group  of  men,  “married”  describes  another. 

Replace  the  [NP  +premod]  in  your  chosen  sentence  pattern  with  the  noun  phrase: 

[[Article]  [Adjective]  and  [Article]  [Adjective]]  [Noun]  ([Postmodification]) 

Eg,  List  [the  single  and  the  married  men  in  the  New  York  branch] . 
Eg,  [The  Guatemalan  and  the  Honduran  terrorists]  bombed  the 


VIII-7 


Government  Houses. 

Eg,  [The  armed  and  the  unarmed  guerrilla  units]  attacked  the 
Government  House. 

Eg,  [The  American  and  the  British  Embassies]  were  attacked 
by  guerrillas. 


Criteria:  Demonstrated  understanding  of  the  qualification  of  the  premodified  noun  phrase 
by  the  conjoined  “[Article]  [Adjective]”  phrases. 

_ Score: _ 


1.1. 4.3  Conjunction  of  Genitive  Noun  Phrases 

Replace  the  [NP  +premod]  in  your  chosen  sentence  pattern  with  the  noun  phrase: 
[[NP  +genitive]  and  [NP  +genitive]|  [Noun] 


Eg.  List  [John  Smith’s  and  Jane  Jones'  salaries]. 

Eg,  What  were  [the  printer’s  and  the  CPU's  sales  in  1990]? 

Eg,  Terrorists  attacked  [the  Prime  Minister’s  and  the  Secretary 
General’s  residences] . 

Eg,  Guerrillas  burned  [the  President’s  and  the  Prime  Minister’s 
offices] . 

Criteria:  Demonstrated  understanding  of  the  qualification  of  the  premodified  noun  phrase 
by  the  conjoined  genitive  noun  phrases. 

_  Score:  _ 


1.1. 4.4  Scope  of  Adjectives  over  Nouns 
A  noun  phrase  which  follows  the  pattern: 

[Adjective]  [Noun]  and  [Noun] 

is  ambiguous,  with  the  possibilities  being  that  either  the  first  noun  only  is  within  the  scope 
of  the  adjective,  or  that  both  nouns  are  within  the  scope  of  the  adjective.  The  ambiguity 
must  be  resolved  using  situational  or  world  knowledge.  Generally,  if  the  two  nouns  refer  to 
subcategories  of  the  same  type  of  thing,  they  will  both  be  modified  by  the  adjective.  For 
example,  in  the  phrase 

single  men  and  women 
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the  typical  interpretation  would  be  that  the  phrase  refers  both  to  men  and  women  who  are 
single,  since  men  and  women  are  subcategories  of  the  type  human. 

In  examples  where  the  two  nouns  refer  to  different  types  of  things,  it  may  not  make  sense 
for  the  adjective  to  modify  both.  For  example,  in  the  phrase 

single  men  and  machinery 

“single”  does  not  modify  “machinery”. 

In  these  tests  we  see  whether  the  system  can  distinguish  the  scope  of  the  adjective  over  the 
nouns  in  the  conjunction. 


1.1. 4. 4.1  Scope  of  Adjective  Includes  First  Noun  Only 

Replace  the  [NP  +premod]  in  your  chosen  sentence  pattern  with  the  noun  phrase: 

[  [Article]  [[Adjective]  [Noun]]  and  [Noun]  ]  ([Postmodification]) 

Choose  an  adjective  and  two  nouns  such  that  the  second  noun  is  not  contained  within  the 
scope  of  the  adjective. 

Eg,  List  [the  single  men  and  salaries  in  the  New  York  branch] . 

Eg,  A  carbomb  killed  [the  Marxist  terrorists  and  President] . 

Eg,  Guerrillas  burned  [the  government  buildings  and  banks] . 

Criteria:  Demonstrated  understanding  of  the  noun  phrase  formed  with  an  adjective  followed 
by  the  conjunction  of  two  nouns,  where  the  first  noun  is  inside  the  scope  of  the  adjective 
and  the  second  noun  is  outside  the  scope  of  the  adjective. 

_  Score:  _ 


1.1. 4. 4. 2  Scope  of  Adjective  Includes  Both  Nouns 

Replace  the  [NP  +premod]  in  your  chosen  sentence  pattern  with  the  noun  phrase: 

[Article]  [Adjective]  [[Noun]  and  [Noun]]  ([Postmodification]) 

Choose  an  adjective  and  two  nouns  such  that  both  conjoined  nouns  are  contained  within  the 
scope  of  the  adjective. 

Eg,  List  [the  single  men  and  women  in  the  New  York  branch] . 

Eg,  [The  Marxist  terrorists  and  guerrillas]  attacked  the  Government 
House. 

Eg,  Guerrillas  burned  [the  government  buildings  and  vehicles] . 
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Criteria:  Demonstrated  understanding  of  the  noun  phrase  formed  with  an  adjective  followed 
by  the  conjunction  of  two  nouns,  where  both  nouns  are  inside  the  scope  of  the  adjective. 

_ Score: _ 


1.1.5  Pronouns  Conjoined  by  AND  (t.b.d.) 

1.1.6  Verb  Phrases  with  the  Conjunction  AND 

Choose  one  of  the  sentence  patterns  below  to  use  in  the  following  subsections.  In  each  of 
the  subsections,  a  verb  phrase  construction  with  the  conjunction  “and”  is  specified.  You  will 
replace  the  [VP  +conjunction]  in  the  chosen  pattern  with  the  specified  verb  phrase. 

Pattern:  List  [NP]  [Rel  Pronoun]  [VP  -^conjunction] 

Example:  List  the  women  who  are  young  and  single. 

Example:  List  the  employees  who  live  in  New  York  and  sell. 

Example:  List  the  employees  who  earn  $45,000  and  sell  printers. 

Pattern:  What  ([NP])  [VP  -(-conjunction]  ? 

Example:  What  employees  does  John  Smith  train  and  supervise? 

Example:  What  employees  buy  and  sell  printers? 

Pattern:  [NP]  [VP  +con  junction]. 

Example:  The  terrorists  shot  and  killed  the  Prime  Minister. 

Example:  The  electric  facility  exploded  and  burned. 

Pattern:  [NP]  [Relative  Pronoun]  [VP  -(-conjunction] 

Example:  The  terrorists  who  were  armed  and  unidentified 

Example:  The  rebels  who  were  young  and  militant 

[VP]. 

bombed  the  Government  House, 
burned  the  vehicles. 

Pattern:  [NP]  [Verb]  [Noun]  [Relative  Pronoun] 

Example:  The  terrorists  attacked  the  Prime  Minister  who 

Example:  The  guerillas  attacked  the  train  which 

Example:  Guerrillas  kidnapped  a  civilian  who 
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[VP  -(-conjunction]. 
was  alive  and  unharmed, 
was  derailed  and  damaged, 
was  an  American  and  a  diplomat. 

Brackets  are  used  in  the  specified  verb  phrases  in  the  following  subsections  to  aid  in  the 
determination  of  which  elements  are  being  conjoined.  Also,  brackets  are  used  in  the  examples 
to  identify  the  phrases  which  follow  the  specified  verb  phrase  pattern. 

1.1. 6.1  BE- Verb  Conjoined  by  AND 

1.1. 6. 1.1  BE- Verb  With  Complement  Conjoined  by  AND 

1.1. 6. 1.1.1  Adjectival  Complements: 

Replace  the  [VP  -(-conjunction]  in  your  chosen  sentence  pattern  with  the  verb  phrase: 

[BE- Verb]  [[Adjective]  and  [Adjective]] 


Eg,  List  the  women  who  [are  young  and  single] . 

Eg,  What  employees  [are  educated  and  experienced]? 

Eg,  The  terrorists  who  [were  armed  and  unidentified]  bombed  the 
Government  House. 

Eg,  The  terrorists  attacked  the  Prime  Minister  who  [is  alive  and 
unharmed] . 

Eg,  The  rebels  who  [were  young  and  militant]  burned  the  vehicles. 
Eg,  Guerrillas  attacked  the  train  which  [was  derailed  and 
damaged] . 

Criteria:  Demonstrated  understanding  of  the  copular  verb  phrase  formed  with  a  BE-ve rb 
followed  by  conjoined  adjectival  complements. 

_ Score: _ 


1.1. 6. 1.1. 2  Noun  Phrase  Complements: 

Replace  the  [VP  -(-conjunction]  in  your  chosen  sentence  pattern  with  the  verb  phrase: 

[BE- Verb]  [[NP]  and  [NP]] 

Eg,  List  the  employees  who  [are  women  and  salespersons] . 

Eg,  The  terrorists  who  [were  Indians  and  Whites]  bombed  the 
Government  House. 

Eg,  The  terrorists  attacked  a  man  who  [is  a  Guatemalan  and  a  Banker] . 
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Eg,  Guerrillas  kidnapped  a  civilian  vho  [is  an  American  and  a 
diplomat] . 

Criteria:  Demonstrated  understanding  of  the  copular  verb  phrase  formed  with  a  BE-verb 
followed  by  conjoined  noun  phrase  complements. 

_ Score: _ 


1.1. 6. 1.1. 3  Mixed  Complements 

In  this  test,  use  one  adjectival  complement  and  one  noun  phrase  complement.  Replace  the 
[VP  -{-conjunction]  in  your  chosen  sentence  pattern  with  the  verb  phrase: 

[BE- Verb]  [Complement]  and  [BE- Verb]  [Complement] 

Eg,  List  the  women  who  [are  managers  and  are  single] . 

Eg,  The  terrorists  who  [were  Whites  and  were  armed]  attacked 
the  Government  House. 

Eg,  The  rebels  who  [are  young  and  are  URNG  members]  attacked 
the  Government  House. 

Eg.  The  guerrillas  kidnapped  a  man  who  [is  an  American  and 
was  wounded] . 

Criteria:  Demonstrated  understanding  of  the  copular  verb  phrase  formed  with  a  BE-ve rb 
followed  by  a  noun  phrase  complement  conjoined  with  an  adjectival  complement. 

- Score:  _ 


1.1. 6. 1.2  Passive  Constructions  with  the  Conjunction  AND 
1.1. 6. 1.2.1  Conjunction  of  the  Verb  in  a  Passive  Verb  Phrase 

Replace  the  [VP  -{-conjunction]  in  your  chosen  sentence  pattern  with  the  verb  phrase: 

[BE- Verb]  [[Verb  (-ed)]  ([NP])  and  [Verb  (-ed)]  ([NP])]  (by  [NP]) 

Eg,  List  the  employees  who  [are  paid  $35,000  and  given  benefits]. 
Eg,  What  employees  [are  paid  $35,000  and  given  benefits]? 

Eg,  The  Prime  Minister  [was  kidnapped  and  shot  by  terrorists] . 

Eg,  The  men  who  [were  captured  and  killed]  had  bombed  the 
Government  House. 

Eg,  The  presidential  farm  [was  attacked  and  burned  by  the 
guerrillas] . 

Eg,  The  buildings  which  [were  bombed  and  destroyed]  were 
government  buildings. 
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Criteria:  Demonstrated  understanding  of  the  passive  verb  phrase  formed  with  the  auxiliary 
BE-ve rb  followed  by  two  conjoined  passive  verbs  (with  optional  objects). 

— — — - - — — - - - - - Score: _ 


1.1. 6. 1.2.2  Conjunction  of  a  Passive  Verb  Phrase 

Replace  the  [VP  -(-conjunction]  in  your  chosen  sentence  pattern  with  the  verb  phrase: 

[BE- Verb]  [Verb  (-ed)]  ([NP])  and  [BE- Verb]  [Verb  (-ed)J  ([NP])  (by  [NP]) 

Eg,  List  the  employees  who  [are  paid  $35,000  and  are  given  benefits]. 
Eg,  What  employees  [are  paid  $35,000  and  are  given  benefits]? 

Eg,  The  Prime  Minister  [was  kidnapped  and  was  shot  by  terrorists] . 

Eg,  The  men  who  [were  captured  and  were  killed]  had  bombed  the 
Government  House. 

Eg,  The  Santo  Tomas  farm  [was  burned  and  was  destroyed  by  guerrillas] 
Eg,  An  American  diplomat  [was  kidnapped  and  was  killed  by  guerrillas] 

Criteria:  Demonstrated  understanding  of  the  passive  verb  phrase  formed  by  the  conjunction 
of  two  passive  verb  phrases. 

- — - - - - . - - - - - Score:  _ 


1.1. 6.2  DO- Verb  Constructions  with  the  Conjunction  AND 

Test  a  DO-verb  as  an  auxiliary  to  a  verb  phrase  formed  by  conjoining  two  verbs  with  “and”. 
Replace  the  [VP  -(-conjunction]  in  your  chosen  sentence  pattern  with  either  the  verb  phrase: 

[DO- Verb]  [[Verb]  and  [Verb]]  ([NP]) 

in  an  declarative  sentence,  or 

[DO- Verb]  [NP]  [[Verb]  and  [Verb]]  ([NP]) 

in  an  interrogative. 

Eg,  What  employees  [does  John  Smith  train  and  supervise]? 

Eg,  The  terrorists  [did  shoot  and  kill  the  Prime  Minister] . 

Eg,  The  guerrillas  [did  kidnap  and  kill  the  American  diplomat] . 

Criteria:  Demonstrated  understanding  of  the  verb  phrase  formed  with  the  auxiliary  DO-verb 
followed  (not  necessarily  immediately)  by  two  conjoined  verbs. 
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1.1. 6. 3  Full  Verb  Constructions  with  the  Conjunction  AND 


Score: 


1.1. 6.3.1  Conjunction  of  Intransitive  Verbs 

Replace  the  [VP  -(-conjunction]  in  your  chosen  sentence  pattern  with  the  verb  phrase: 
[Verb  +intrans]  and  [Verb  +intrans] 

Eg,  What  employees  [live  and  work]  in  New  York? 

Eg,  The  electric  facility  [exploded  and  burned] . 

Eg,  (Preliminary  Sentence:  The  guerrillas  shot  a  civilian.) 
The  civilian  [suffered  and  died] . 


Criteria:  Demonstrated  understanding  of  the  verb  phrase  formed  with  the  conjunction  of 
two  intransitive  verbs. 


_ _  Score: _ 

1.1. 6. 3. 2  Conjunction  of  Transitive  Verbs 

1.1. 6. 3. 2.1  Conjunction  of  Transitive  Verb  Phrases 

Replace  the  [VP  -(-conjunction]  in  your  chosen  sentence  pattern  with  the  verb  phrase: 
[Verb  +trans]  [NP]  and  [Verb  4-trans]  [NP] 

Eg.  List  the  employees  who  [earn  $45,000  and  sell  printers]. 

Eg,  Terrorists  [bombed  the  Government  House  and  killed  the  Prime 
Minister] . 

Eg,  Guerrillas  [attacked  the  presidential  farm  and  killed  a 
civilian] . 

Eg,  The  rebels  [burned  vehicles  and  robbed  shops] . 


Criteria:  Demonstrated  understanding  of  the  verb  phrase  formed  with  the  conjunction  of 
two  transitive  verbs,  each  with  an  object. 

- - Score: _ 


1.1. 6. 3. 2.2  Conjunction  of  Two  Transitive  Verbs  with  Single  Object  NP 

Replace  the  [VP  -(-conjunction]  in  your  chosen  sentence  pattern  with  the  verb  phrase: 


VIII- 14 


[[Verb  +trans]  and  [Verb  +trans]]  [NP] 

Eg,  What  company  [manufactures  and  sells  printers]? 

Eg,  Terrorists  [shot  and  killed  the  Prime  Minister] . 

Eg,  The  guerrillas  [attacked  and  burned  the  presidential  farm] . 
Eg,  The  guerrillas  [bombed  and  destroyed  the  Central  Bank] . 


Criteria:  Demonstrated  understanding  of  the  verb  phrase  formed  with  the  conjunction  of 
two  transitive  verbs  which  share  the  object  following  the  second  verb. 

_ Score: _ 


1.1. 6.4  Mixed  Verb  Phrases  in  Constructions  with  AND 

1.1. 6. 4.1  Conjunction  of  Transitive  and  Intransitive  Verbs 

There  is  some  difficulty  in  interpreting  the  conjunction  of  intransitive  and  transitive  verbs. 
When  the  transitive,  followed  by  its  object,  precedes  the  intransitive,  it  is  clear  that  the 
latter  verb  is  intransitive,  since  there  is  no  following  noun  phrase  to  function  as  the  object 
of  the  second  verb.  For  example,  in  the  sentence: 

Smith  sells  printers  and  purchases. 

it  is  clear  that  selling  printers  is  one  of  Smith’s  tasks,  and  purchases  is  the  other.  When  the 
intransitive  precedes  the  transitive  it  is  often  possible  to  erroneously  interpret  the  first  verb 
as  being  transitive  and  thereby  sharing  the  final  noun  phrase.  A  likely  interpretation  of  the 
sentence: 

Smith  purchases  and  sells  printers. 

is  that  one  of  Smith’s  two  tasks  is  purchasing  printers. 

1.1. 6. 4. 1.1  Conjunction  of  Transitive  with  Intransitive 

Replace  the  [VP  -4-conjunction]  in  your  chosen  sentence  pattern  with  the  verb  phrase: 

[Verb  4-trans]  [NP]  and  [Verb  -f  INTRANs] 

Eg,  List  the  employees  who  [sell  printers  and  live  in  New  York] . 
Eg,  Terrorists  [bombed  the  Government  House  and  died] . 

Eg,  Two  ARENA  members  [shot  the  American  Ambassador  and  died 
during  the  ambush] . 


VIII- 15 


Criteria:  Demonstrated  understanding  of  the  verb  phrase  formed  with  a  transitive  verb  and 
its  object  conjoined  with  an  intransitive  verb. 

_ Score:  _ 


1.1. 6. 4. 1.2  Conjunction  of  Intransitive  with  Transitive 

Replace  the  [VP  -1-conjunction]  in  your  chosen  sentence  pattern  with  the  verb  phrase: 

[Verb  +intrans]  and  [Verb  +trans]  [NP] 

Eg,  What  salesperson  [works  and  sells  printers]? 

Eg,  A  car  bomb  [exploded  and  killed  twenty  peasants] . 

Criteria:  Demonstrated  understanding  of  the  verb  phrase  formed  with  an  intransitive  verb 
conjoined  with  a  transitive  verb  followed  by  its  object. 

_ Score:  _ 


1.1. 6. 4. 2  Conjunction  of  Copular  with  Non-Copular  Active  Verb  Phrase 
1.1. 6. 4. 2.1  With  Copular  First 

Replace  the  [VP  +conjunction]  in  your  chosen  sentence  pattern  with  the  following  verb 
phrase,  where  [VP]  is  any  non-copular  verb  phrase: 

[BE- Verb]  [Complement]  and  [VP] 

Eg,  What  employees  [are  women  and  earn  $45,000]? 

Eg,  The  terrorists  [are  assassins  and  killed  the  Prime  Minister] . 
Eg,  The  terrorists  [are  Marxists  and  killed  the  Prime  Minister] . 
Eg,  The  rebels  [were  armed  and  killed  two  American  citizens] . 


Criteria:  Demonstrated  understanding  of  the  verb  phrase  formed  by  the  conjunction  of  a 
copular  verb  phrase  with  a  non-copular  verb  phrase. 

- Score:  _ 


1.1. 6. 4. 2. 2  With  Non-Copular  First 

Replace  the  [VP  +conjunction]  in  your  chosen  sentence  pattern  with  the  following  verb 
phrase,  where  [VP]  is  any  non-copular  verb  phrase: 
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[VP]  and  [BE- Verb]  [Complement] 


Eg,  What  employees  [earn  $45,000  and  are  women]? 

Eg,  The  terrorists  [killed  the  Prime  Minister  and  are  assassins] . 
Eg,  The  terrorists  [killed  the  Prime  Minister  and  are  Marxists] . 
Eg,  A  guerrilla  [bombed  the  Central  Bank  and  was  wounded] . 

Criteria:  Demonstrated  understanding  of  the  verb  phrase  formed  by  the  conjunction  of  a 
non-copular  verb  phrase  with  a  copular  verb  phrase. 

_ Score:  _ 


1.1. 6. 4. 3  Conjunction  of  Passive  with  Non-Copular  Active  Verb  Phrase 
1.1. 6. 4. 3.1  With  Passive  First 

Replace  the  [VP  +  conjunction]  in  your  chosen  sentence  pattern  with  the  following  verb 
phrase,  where  [VP]  is  a  verb  phrase  in  active  voice: 

[[BE- Verb]  [Verb  (-ed)]  ([NP])  (by  [NP])]  and  [VP] 

Eg.  List  the  employees  who  [are  supervised  and  earn  $25,000]. 

Eg,  List  the  employees  who  [are  paid  $50,000  and  manage  a 
department] . 

Eg,  The  Prime  Minister  [was  shot  by  terrorists  and  died] . 

Eg,  An  American  diplomat  [was  kidnapped  by  guerrillas  and  died] . 
Eg,  A  tanker  [was  bombed  by  terrorists  and  exploded] . 

Criteria:  Demonstrated  understanding  of  the  verb  phrase  formed  by  the  conjunction  of  a 
passive  verb  phrase  with  a  (non-copular)  active  verb  phrase. 

_ Score: _ 


1.1. 6. 4. 3. 2  With  Non-Copular  Active  Verb  Phrase  First 

Replace  the  [VP  -(-conjunction]  in  your  chosen  sentence  pattern  with  the  following  verb 
phrase,  where  [VP]  is  a  verb  phrase  in  active  voice: 

[VP]  and  [[BE- Verb]  [Verb  (-ed)]  ([NP])  (by  [NP])] 

Eg,  List  the  employees  who  [earn  $25,000  and  are  supervised]. 

Eg,  List  the  employees  who  [manage  a  department  and  are  paid  $50,000] . 
Eg,  The  terrorists  [attacked  the  Government  House  and  were  killed] . 

Eg,  A  tanker  [anchored  and  was  attacked  by  the  guerrillas] . 
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Criteria:  Demonstrated  understanding  of  the  verb  phrase  formed  by  the  conjunction  of  a 
(non-copular)  active  verb  phrase  with  a  passive  verb  phrase. 

_ _ Score: _ 


1.1. 6. 4. 4  Conjunction  of  Copular  with  Passive 
1.1. 6. 4. 4.1  With  Copular  First 

Replace  the  [VP  -1-conjunction]  in  your  chosen  sentence  pattern  with  the  verb  phrase: 

[BE- Verb]  [Complement]  and  [BE- Verb]  [Verb  (-ed)]  ([NP])  (by  [NP]) 

Eg,  List  the  employees  who  [are  women  and  are  paid  $50,000]. 

Eg,  The  Prime  Minister  who  [is  a  Guatemalan  and  was  shot] ,  died. 
Eg,  The  man  who  [is  the  Prime  Minister  and  was  shot],  died. 

Criteria:  Demonstrated  understanding  of  the  verb  phrase  formed  by  the  conjunction  of  a 
copular  verb  phrase  with  a  passive  verb  phrase. 

_ Score:  _ 


1.1. 6. 4. 4. 2  With  Passive  First 

Replace  the  [VP  -{-conjunction]  in  your  chosen  sentence  pattern  with  the  verb  phrase: 

[BE- Verb]  [Verb  (-ed)]  ([NP])  (by  [NP])  and  [BE- Verb]  [Complement] 

Eg,  List  the  employees  who  [are  paid  $50,000  and  are  women]. 

Eg,  The  Prime  Minister  who  [was  shot  and  was  a  Union  leader],  died. 
Eg,  Guerrillas  killed  an  American  who  [was  a  diplomat  and  was 
kidnapped] . 

Criteria:  Demonstrated  understanding  of  the  verb  phrase  formed  by  the  conjunction  of  a 
passive  verb  phrase  with  a  copular  verb  phrase. 

_ Score: _ 


1.1.7  Conjunction  of  Adverbials 

Test  two  adverbials  conjoined  with  “and”. 

Pattern:  List  [NP]  [Rel  Pronoun]  [VP]  [[Adverbial]  and 

Example:  List  the  salespersons  who  earned  $35,000  in  1989  and 
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[Adverbial]]? 
in  1990. 

Pattern:  What  [NP]  [VP]  [[Adverbial]  and  [Adverbial]]? 

Example:  What  salespersons  earned  $35,000  in  1989  and  in  1990? 

Pattern:  [NP]  [VP]  [[Adverbial]  and  [Adverbial]]. 

Example:  Terrorists  attacked  the  Government  House  yesterday  and  on  January  10. 

Example:  Guerrillas  bombed  the  presidential  farm  yesterday  and  today. 

Criteria:  Demonstrated  understanding  of  the  conjunction  of  the  two  adverbials. 

_ _ _  Score:  _ _ 


1.1.8  Conjunction  within  Prepositional  Phrases 

Remember  that  a  prepositional  phrase  consists  of  a  preposition  followed  by  a  complement, 
which  is  typically  a  noun  phrase,  for  example:  “in  the  department”,  “for  Joe  Smith”,  “on 
December  12”. 

In  order  to  keep  the  following  tests  as  simple  as  possible,  use  simple  noun  phrases  within  the 
prepositional  phrases.  That  is,  avoid  noun  phrases  which  contain  postmodifiers. 


1.1. 8.1  Conjunction  of  Prepositional  Phrases 

Test  two  prepositional  phrases  conjoined  with  “and”. 

Pattern:  List  [NP]  [[PP]  and  [PP]]. 

Example:  List  the  employees  in  the  New  York  branch  and  in  the  Chicago  branch. 


Pattern: 

What  ([NP]) 

[BE- Verb] 

[NP] 

[[PP] 

and  [PP]] 

[Verb  (-c 

Pattern: 

What  ([NP]) 

[DO-Verb] 

[NP] 

[[PP] 

and  [PP]] 

[Verb]  ? 

Pattern: 

[NP] 

[[PP] 

and 

[pp]] 

Example: 

Terrorists 

from  Guatemala 

and 

from  Honduras 

Example: 

The  guerrillas 

in  Guatemala  City  and 

in  Quezaltenango 

[VP]. 

shot  the  Prime  Ministers, 
bombed  electric  power  stations. 


Pattern:  [NP]  [Verb]  [NP]  [[PP]  and  [PP]]. 

Example:  Terrorists  bombed  the  Government  Houses  in  Honduras  and  in  Guatemala. 
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Criteria:  Demonstrated  understanding  of  the  conjunction  of  the  two  prepositional  phrases. 
_ _  Score: _ 


1.1. 8. 2  Conjunction  of  Noun  Phrases  within  a  Prepositional  Phrase 

Test  two  noun  phrases  conjoined  with  “and”  within  a  prepositional  phrase. 

Pattern:  List  [NPj  [Preposition]  [[NP]  and  [NP]] . 

Example:  List  the  employees  in  the  New  York  and  Chicago  branches. 

Pattern:  What  ([NP])  [BE- Verb]  [NP]  [Preposition]  [[NP]  and  [NP]]  [Verb  (-ed)]  ? 

Pattern:  What  ([NP])  [DO-Verb]  [NP]  [Preposition]  [[NP]  and  [NP]]  [Verb]  ? 

Pattern:  [NP]  [Preposition]  [[NP]  and  [NP]] 

Example:  Terrorists  from  Guatemala  and  Honduras 

Example:  Guerrillas  in  Guatemala  City  and  Quezaltenango 

[VP]. 

shot  the  Prime  Ministers, 
bombed  electric  power  stations. 

Pattern:  [NP]  [Verb]  [NP]  [Preposition]  [[NP]  and 

Example:  Terrorists  bombed  the  Government  House  in  November  and 

Example:  Guerrillas  attacked  the  presidential  farm  on  January  16  and 

(NP]]. 

December. 

February  3. 

Criteria:  Demonstrated  understanding  of  the  prepositional  phrase  formed  with  a  preposition 
followed  by  two  conjoined  noun  phrases. 

_ Score: _ 


1.1. 8. 3  Conjunction  of  Prepositions  within  a  Prepositional  Phrase 

Test  two  prepositions  conjoined  with  “and”  within  a  prepositional  phrase. 

Pattern:  List  [NP]  [[Preposition]  and  [Preposition]]  [NP]. 

Example:  List  the  personnel  above  and  below  the  manager  in 

the  New  York  branch. 
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Pattern:  What  ([NP])  [BE-Verb]  [NP]  [[Preposition]  and  [Preposition]]  [NP] 

[Verb  (-ed)]  ? 

Pattern:  What  ([NP])  [DO-Verb]  [NP]  [[Preposition]  and  [Preposition]]  [NP] 


[Verb]  ? 

Pattern: 

[NP] 

[[Preposition] 

and 

[Preposition]] 

Example: 

Power  transmission  towers  to 

and 

from 

Example: 

The  houses 

in 

and 

around 

[NP] 

[VP]- 

the  electric  facility 

were  bombed. 

Guatemala  City 

were  burned  by  the  URNG  on  February  1. 

Pattern: 

[NP]  [Verb] 

[NP] 

[[Preposition] 

and 

[Preposition]] 

Example: 

Terrorists  bombed 

the  electric  facilities 

in 

and 

around 

Example: 

The  guerrillas  killed 

civilians 

in 

and 

around 

[NP]. 

Honduras. 
Guatemala  City. 

Criteria:  Demonstrated  understanding  of  the  prepositional  phrase  formed  with  the  conjunc¬ 
tion  of  two  prepositions  followed  by  a  noun  phrase. 

_ _ Score: _ 


1.2  BUT  as  a  Conjunction  (t.b.d.) 

1.3  OR  as  a  Conjunction  (t.b.d.) 

1.4  Multiple  Coordination  (t.b.d.) 

2  Subordinators  (t.b.d.) 

3  Conjuncts  (t.b.d.) 

4  Subjuncts  (t.b.d.) 
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IX.  EMBEDDED  SENTENCES 


A  complex  sentence  consists  of  a  main  or  superordinate  sentence  (clause)  with  an  embedded 
sentence  (clause)  called  the  subordinate  sentence  (clause).  In  this  Procedure,  the  sections 
on  relative  clauses  and  comparatives  have  included  certain  types  of  embedded  clauses.  Em¬ 
bedded  or  subordinate  clauses  may  function  as  subject,  object,  complement,  or  adverbial  in 
a  superordinate  sentence  (clause).  In  the  following  list  of  examples,  the  embedded  sentence 
or  clause  is  enclosed  in  brackets  for  ease  of  identification. 

Subject:  [ That  the  Personnel  Department  works  a  4-day  week]  is  unacceptable  to  the  other 
employees. 

Direct  object:  The  employees  in  Department  A  preferred  [to  retrain]. 

Indirect  object:  You  can  tell  [whoever  is  waiting]  that  I’ll  be  back  in  ten  minutes. 
Subject  complement:  One  result  of  layoffs  is  [that  morale  will  go  down]. 

Object  complement:  I  know  her  [to  be  reliable]. 

Adverbial:  [When  John  reaches  $1,000,000  in  sales],  he  will  be  promoted. 

Postmodifier  in  a  noun  phrase  (relative  clause):  The  previous  V.P.  [that  was  in  charge 
of  the  Systems  Division]  retired  in  1985. 

Prepositional  complement:  John’s  promotion  depends  on  [whether  he  reaches  $1,000,000 
in  sales]. 

Adjective  complement:  It  is  true  [that  John  Smith  is  V.P.  of  Finance ]. 

This  section  covers  nominal  embedded  clauses  and  we  defer  coverage  of  the  other  types  of 
embedded  clauses  to  a  subsequent  version  of  the  Procedure.  Some  other  types  of  embedded 
clauses  are  covered  in  Section  XII,  Semantics  of  Events. 

Nominal  clauses  aie  clauses  that  function  as  noun  phrases.  The  following  subsections  cover 
different  types  of  nominal  clauses.  In  each  of  the  following  subsections,  use  sentence  patterns 
which  the  system  has  successfully  processed  and  substitute  an  embedded  clause  (as  described 
in  the  subsection)  for  a  noun  phrase  in  the  basic  sentence  pattern. 

In  the  examples  in  the  following  subsections,  the  embedded  clauses  are  enclosed  in  brackets 
for  ease  of  identification. 

1.  THAT-Cl&uses 

A  that- clause  consists  of  the  word  “that”  followed  by  a  complete  sentence.  The  that- clause  is 
embedded  within  a  superordinate  sentence  as  a  nominal.  In  each  of  the  following  examples, 
the  that- clause  functioning  as  a  nominal  (in  the  place  of  a  noun  phrase)  is  enclosed  in  brackets. 

Test  a  that-c\&use  as  an  embedded  nominal. 
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Eg,  Is  it  true  [that  John  Smith  earns  $30,000]? 

Eg,  Is  it  the  case  [that  Mary  Jones  works  in  the  Chicago  branch]? 

Eg,  John  was  informed  [that  he  will  travel  to  the  Boston  office 
next  week] . 

Eg,  Government  officials  reported  [that  terrorists  bombed  the 
government  house  in  Andahua] . 

Eg,  Investigators  announced  [that  the  mayor  was  murdered  by 
terrorists] . 

Eg,  The  FMLN  claimed  [that  FMLN  members  bombed  the  National 
Bank] . 

Eg,  [That  the  URNG  guerrillas  attacked  the  presidential  farm] 
was  reported  in  the  newspaper. 

Eg,  The  guerrillas  claimed  [that  they  had  kidnapped  the  mayor] . 


Criteria:  Demonstrated  understanding  of  the  embedded  sentence  (THAT-clause)  in  the  con¬ 
text  of  the  superordinate  sentence  (the  sentence  in  which  it  is  embedded). 

_ _ _ _ Score: _ 


2.  Wh-interrogative  clauses 

A  Wh-Interrogative  clause  is  a  sentence  that  begins  with  a  WH-word  such  as  what ,  who, 
which,  why ,  etc. 

Test  a  Wh-interrogative  clause  as  an  embedded  nominal. 

Eg,  Tell  me  [who  is  the  head  of  the  sales  department] . 

Eg,  Tell  me  [which  employees  are  members  of  the  sales  department] . 
Eg,  Tell  me  [when  John  Smith  was  hired] . 

Eg,  Authorities  are  trying  to  determine  [who  bombed  the 
National  Bank] . 

Eg,  The  government  has  not  determined  [who  attacked  the  presidential 
farm] . 

Eg,  February  4  was  [when  guerrillas  attacked  the  presidential  farm] . 

Criteria:  Demonstrated  understanding  of  the  embedded  sentence  (Wh-interrogative  clause) 
in  the  context  of  the  superordinate  sentence  (the  sentence  in  which  it  is  embedded). 

- — — - - — - _ Score: _ 

3.  Yes-No  interrogative  clauses 

A  Yes-no  interrogative  clause  is  introduced  by  whether  or  if.  The  sentence  following  the  word 
whether  or  if  is  either  true  or  false. 

Test  a  Yes-No  interrogative  clause  as  an  embedded  nominal. 
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Eg,  Tell  me  [whether  John  Smith  is  a  salesperson] . 

Eg,  Tell  me  [if  Ted  Harris  is  a  Senior  Engineer] . 

Eg,  Tell  me  [if  Mary  Jones  earned  $30,000  in  1989]. 

Eg,  State  [whether  John  Smith  works  in  Boston] . 

Eg,  The  authorities  are  investigating  [whether  the  bombing  of  the 
bank  was  done  by  the  FMLN] . 

Eg,  It  is  not  clear  [whether  the  damage  caused  by  the  guerrilla 
attack  on  the  presidential  farm  was  severe] . 

Eg,  It  is  unknown  [whether  the  American  citizen  kidnapped  by  the 
the  guerrillas  is  still  alive] . 


Criteria:  Demonstrated  understanding  of  the  embedded  sentence  (Yes-No  interrogative  clause) 
in  the  context  of  the  super  ordinate  sentence  (the  sentence  in  which  it  is  embedded). 

_ Score: _ 


4.  Infinitive  clauses  introduced  by  TO 

A  TO-infinitive  clause  consists  of  the  work  TO  followed  by  a  verb  phrase. 

Test  a  TO-infinitive  clause  as  an  embedded  nominal. 

Eg,  Who  decided  [to  work  in  the  Chicago  branch]? 

Eg,  Who  does  Xidec  employ  [to  manage  the  sales  department]? 

Eg,  Which  employees  were  able  [to  earn  $40,000  in  1989]? 

Eg,  The  terrorists  claimed  [to  have  kidnapped  three  cabinet 
members] . 

Eg,  Terrorists  were  reported  [to  have  kidnapped  the  mayor 
of  Bogota] . 

Eg,  The  guerrillas  attempted  [to  bomb  the  Central  Bank  this 
morning] . 


Criteria:  Demonstrated  understanding  of  the  embedded  sentence  (TO-infinitive  clause)  in 
the  context  of  the  superordinate  sentence  (the  sentence  in  which  it  is  embedded). 

_ Score: _ 


5.  -ING  clauses 

An  -ING  clause  consists  of  an  -ING  participle  used  as  the  head  of  a  noun  phrase  with 
optional  premodification  and  postmodification. 

Test  an  -ING  clause  as  an  embedded  nominal. 


Eg,  Is  [earning  $50,000]  an  attribute  of  John  Smith? 

Eg,  Is  [working  in  Boston]  a  characteristic  of  Mary  Jones? 
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Eg,  [Being  50  years  old]  makes  someone  "a  senior  employee". 

Eg,  [Bombing  the  bank]  vas  the  work  of  the  FMLN. 

Eg,  The  FMLN  are  claiming  credit  for  [kidnapping  the  mayor] . 

Eg,  [Attacking  the  presidential  farm]  was  carried  out  by  the  URNG 
guerrillas. 

Eg,  The  URNG  guerrillas  claimed  credit  for  [burning  the  presidential 
farm] . 


Criteria:  Demonstrated  understanding  of  the  embedded  sentence  (-ING  clause)  in  the  context 
of  the  superordinate  sentence  (the  sentence  in  which  it  is  embedded). 

- - - - - - - Score: _ 
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X.  REFERENCE 


There  axe  many  ways  of  refering  to  an  entity,  concept,  situation,  or  proposition.  For  example, 
Luis  Carlos  Galan,  a  well-known  victim  of  the  drug  war  in  Colombia,  could  be  referred  to 
as: 

Luis  Carlos  Galan , 

a  well-known  victim  of  the  drug  war  in  Colombia , 

Luis  Carlos  Galan  Sarmiento, 

Senator  Galan , 
presidential  candidate , 
presidential  hopeful, 
liberal  senator, 
murder  victim, 

the  Colombian  government  official  assassinated  on  August  18,  1989 

among  many  other  possibilities.  All  these  words  and  phrases,  as  well  as  others  too  numerous 
to  list,  represent  a  subset  of  the  set  of  referring  expressions  indicating  the  same  referent  in 
the  real  world,  i.e.,  the  unfortunate  Senator  Galan  of  Colombia.  Since  these  expressions  refer 
to  the  same  referent,  they  are  called  co-referential. 

References  may  be  categorized  as  specific  or  generic.  Generally,  a  reference  is  called  specific 
when  it  refers  to  a  particular  object,  and  generic  if  it  refers  to  a  class  or  type  of  object, 
without  specific  reference  to  one  or  more  particular  objects  within  the  class  or  type. 

1.  Specific  Reference 

1.1  Anaphoric  Reference 

“The  term  ANAPHORIC  REFERENCE  is  used  where  the  uniqueness  of  reference  of  some 
phrase  the  X  is  supplied  by  information  given  earlier  in  the  discourse.”  (Quirk  et.  al.,  1985,  p. 
267)  Typically,  a  fairly  explicit  refering  expression  brings  an  object  into  the  discourse  focus 
and  a  subsequent  terse  and  less  explicit  expression  may  refer  to  the  same  object  (and  they  are 
thereby  co-referential).  The  initial  refering  expression  is  called  the  antecedent  expression  and 
the  subsequent  expression  that  refers  to  the  same  object  is  called  the  anaphoric  expression. 
The  anaphoric  expression  “refers  to  the  referent  of  the  antecedent  expression  with  which  it 
is  correlated”  (Lyons,  1977,  p.  660). 

The  expression  which  provides  the  referent  of  an  anaphoric  expression  must  precede  the 
anaphoric  expression  in  the  discourse.  If  it  is  a  constituent  of  the  same  sentence,  the  type 
of  anaphora  is  called  intra-sentential  anaphora,  and  if  it  is  a  constituent  of  some  preceding 
sentence,  the  type  of  anaphora  is  called  inter- sentential  anaphora.  The  following  sections  on 
anaphora  have  included  both.  We  include  one  specific  subsection  that  tests  for  each  type 
explicitly. 

In  the  following  example,  the  phrase  “Shining  Path  guerrillas ”  is  the  antecedent  expression 
for  the  anaphoric  expression  “they”.  In  this  and  following  examples,  brackets  are  added  to 
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aid  the  reader  in  identifying  the  expressions  of  interest. 

[Shining  Path  guerrillas]  appeared  in  Andahua,  where  [they]  murdered  the  mayor. 

Anaphoric  expressions  are  commonly  pronouns,  as  in  the  above  example,  or  definite  noun 
phrases  (i.e.,  a  noun  phrase  with  the  definite  article  “the”).  In  the  following  example,  the 
phrase  “Shining  Path  guerrillas” is  the  antecedent  expression  and  the  phrase  “the  guerrillas” 
is  the  anaphoric  expression. 

On  January  5,  [Shining  Path  guerrillas]  entered  Andahua  and  murdered  the 
mayor.  [ The  guerrillas]  were  apprehended  and  jailed  the  next  day. 

A  sentence  or  clause  describing  an  event  can  also  be  the  antecedent  expression  for  an 
anaphoric  expression.  In  the  following  example,  the  first  sentence  is  the  antecedent  ex¬ 
pression  for  the  anaphoric  reference  “the  assassination”  in  the  second  sentence: 

On  August  18,  1989,  Senator  Galan  was  assassinated  in  Bogota.  [The  assassi¬ 
nation]  was  attributed  to  a  group  called  the  Sinister  Extraditables. 

Procedures  for  testing  these  and  other  types  of  anaphora,  such  as  anaphoric  references  gen¬ 
erated  by  the  pro-predicates  “so”  and  “as”  are  given  in  the  following  subsections.  Suggested 
patterns  and  examples  are  provided.  You  may  use  different  sentence  types  with  which  the 
system  has  been  successful,  as  long  as  you  incorporate  a  test  for  anaphora  that  fits  the 
classification  or  specifications  of  each  subsection. 

Since  there  is  great  variety  in  the  types  of  anaphora,  it  is  unfeasible  to  test  all  forms  or  types 
of  anaphora  in  the  Benchmark  Procedure,  especially  all  the  anaphoric  devices  which  involve 
vaguely  specified  sets  of  propositions.  For  this  reason,  the  procedure  for  evaluating  anaphora 
will  be  limited  to  the  common  anaphoric  phenomena  that  involve  relatively  direct  reference 
and  are  testable  in  a  straightforward  manner. 

1.1.1  Pronominal  Anaphora 

Pronominal  reference  is  the  classic  case  of  anaphora,  as  in  the  sentence: 

[The  Human  Rights  Commission  of  the  National  Dialog,  which  emerged  from  the 
Esquipulas  Peace  Agreement],  expressed  [its]  concern. 

In  this  sentence,  “ its ”,  a  possessive  pronoun,  is  an  example  of  pronominal  anaphora,  which 
refers  to  the  referent  of  the  expression  “The  Human  Rights  Commission  of  the  National 
Dialog,  which  emerged  from  the  Esquipulas  Peace  Agreement”. 

In  the  following  subsections,  the  anaphoric  pronoun  is  represented  as  [NP  +pro],  indicating 
a  noun  phrase  that  is  realized  as  a  pronoun,  and  the  antecedent  noun  phrase  or  sentence  is 
labeled. 
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1.1. 1.1  Anaphoric  References  With  NP  Antecedents 

1.1. 1.1.1  Anaphoric  Reference  Functions  as  Subject 

This  subsection  tests  the  use  of  the  anaphoric  reference  (pronoun)  as  the  subject  of  a  sentence. 
Discussion  and  examples  of  both  active  and  passive  voice  constructions  are  included.  You 
are  to  create  an  input  to  the  NLP  system  that  used  either  voice.  Be  sure  that  the  anaphoric 
reference  that  you  use  is  NOT  ambiguous.  A  reference  is  ambiguous  if  there  is  more  than 
one  meaning  (interpretation  or  referent)  that  can  be  assigned  to  the  expression  or  word. 
Ambiguous  references  are  covered  in  a  later  subsection. 

ACTIVE  CONSTRUCTION.  (Anaphoric  reference  functions  as  subject  of  active  verb) 

In  this  first  example,  the  anaphoric  reference  is  the  pronoun  “they”  in  the  second  sentence. 
Its  referent  (meaning)  is  the  group  of  employees  mentioned  in  the  first  sentence. 

Which  [Antecedent  NP]  [VP]  ? 

[BE- Verb]  [NP  +  PRO  ]  [NP]  ? 

Eg,  Which  [employees]  earn  over  $5,000  a  month  ? 

Are  [they]  managers  ? 

In  this  second  example,  the  anaphoric  reference  is  the  pronoun  “they”  in  the  second  sentence. 
Its  referent  (meaning)  is  the  group  called  the  Shining  Path  guerrillas,  mentioned  in  the  first 
sentence. 

[Antecedent  NP]  [VP]  [Rel-Pronoun]  [NP  +pho]  [VP]  . 

Eg,  [The  Shining  Path  guerrillas]  appeared  in  Andahua, 
where  [they]  murdered  the  mayor. 

PASSIVE  CONSTRUCTION.  (Anaphoric  reference  functions  as  subject  of  passive  verb) 

[BE- Verb]  [Antecedent  NP]  [Verb  -(-passive]  ([Complement])? 

When  [BE- Verb]  [NP  +pro]  [Veib  -|-passive]  ([Complement])? 

Eg,  Was  John  Smith  hired  in  1989  ? 

When  was  [he]  promoted  ? 

[Antecedent  NP]  [VP]  [Conj  +subord]  [NP  -(-pro  ]  [VP]- 

Eg,  [The  Brazilian  journalist  Juan  Castellar]  was  in  Bogota 
when  [he]  was  kidnapped  by  Colombian  Army  of  National 
Liberation  guerrillas. 
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Criteria:  Correctly  determined  which  entity,  previously  mentioned  in  the  same  or  preceding 
sentence,  is  refered  to  by  the  anaphoric  reference  (pronoun). 


Score: 


1.1. 1.1. 2  Anaphoric  Reference  Functions  as  Object 

This  subsection  tests  the  use  of  the  anaphoric  reference  (pronoun)  as  the  object  of  a  sentence 
(the  object  upon  which  the  action  expressed  by  the  verb  is  carried  out).  When  you  create 
your  input  to  the  NLP  system,  be  sure  that  the  anaphoric  reference  that  you  use  is  NOT 
ambiguous. 

In  this  first  example,  the  anaphoric  reference  is  the  pronoun  “him ”,  used  as  the  object  of 
“hire”  in  the  second  sentence.  Its  referent  (meaning)  is  the  entity  John  Smith  mentioned  in 
the  first  sentence. 

[BE- Verb]  [Antecedent  NP]  ([Complement])? 

When  [DO- Verb]  [NP]  [Verb]  [NP  +pro]  ? 

Eg,  Is  [John  Smith]  in  the  sales  department? 

When  did  Xidec  hire  [him]  ? 

[Antecedent  NP]  [VP]. 

[NP]  [Verb]  [NP  +pro]. 

Eg,  [The  mayor]  was  kidnapped  on  December  12. 

On  December  14,  the  kidnappers  killed  [him] . 

[Antecedent  NP]  [VP]  [Conj  +subord]  [NP]  [Verb]  [NP  +pro]. 

Eg,  [The  Brazilian  journalist  Juan  Castellar]  was  in  Bogota 
to  write  an  article  about  the  drug  war 
when  Colombian  Army  of  National  Liberation  guerrillas 
kidnapped  [him] . 

[Antecedent  NP]  [VP]  [RelPronoun]  [NP]  [Verb]  [NP  +pro]  ([Complement]). 

Eg,  [The  Shining  Path  guerrillas]  appeared  in  Andahua,  where 
the  authorities  accused  [them]  of  murdering  the  mayor. 
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Criteria:  Correctly  determined  which  entity,  previously  mentioned  in  the  same  or  preceding 
sentence,  is  refered  to  by  the  anaphoric  reference  (pronoun)  used  as  an  object. 


Score: 


1.1. 1.1. 3  Anaphoric  Reference  Functions  as  Possessive 

This  section  tests  a  systems’  understanding  of  an  anaphoric  refering  pronoun  used  in  the 
possessive  sense.  The  possessive  pronouns  are  “my”,  “our”,  “your”,  “his”,  “her”,  “its”, 
“their”.  Create  a  test  input  that  mentions  some  entity  and  subsequently  refers  back  to  the 
entity  by  using  a  possessive  pronoun. 

In  the  first  example  below,  the  person  John  Smith  is  mentioned  and  the  possessive  pronoun 
“his”  is  used  to  refer  back  to  this  same  person.  The  other  examples  are  similar. 

Eg,  Did  [John  Smith]  meet  [his]  sales  quota  ? 

Eg,  Did  [Mary  Harris]  meet  [her]  sales  quota  for  July  ? 

Eg,  The  Shining  Path  guerrillas  murdered  [the  mayor]  by  bombing 
[his]  home. 

Eg,  The  Guatemalan  guerrilla  murdered  [the  Bolivian  woman]  in 
[her]  own  country. 

Eg,  The  guerrillas  attacked  [the  president]  at  [his]  farm. 

Eg,  [The  mayor]  blamed  the  URNG  guerrillas  for  [his]  kidnapping. 

Eg,  [The  guerrillas]  reported  [their]  attack  on  the  presidential  farm. 

Criteria:  Correctly  determined  which  entity,  previously  mentioned  in  the  same  or  preceding 
sentence,  is  refered  to  by  the  anaphoric  reference  (possessive  pronoun). 


Score: 


1.1. 1.1. 4  Ambiguous  Anaphoric  References 

An  anaphoric  reference  is  ambiguous  if  there  is  more  than  one  entity  to  which  the  reference 
could  refer.  Gender,  number  (singular  or  plural),  and  recency  of  mention  are  used  to  de¬ 
termine  the  proper  referent  of  an  anaphoric  expression.  You  are  to  create  an  input  for  the 
NLP  system  which  consists  of  one  or  more  sentences.  The  input  must  mention  two  possible 
referents  before  the  anaphoric  use  of  the  pronoun.  The  system  should  use  its  knowledge  of 
the  topic  or  application  domain  in  order  to  determine  the  correct  referent  of  the  pronoun. 

In  the  first  example  below,  there  are  two  candidate  referents  for  the  pronoun  “he”  (in  brack¬ 
ets),  the  FMLN  leader  and  the  Andahua  mayor.  They  are  both  recently  mentioned  and  both 
satisfy  the  gender  and  number  requirements.  To  decide  which  is  the  logical  choice  to  be  the 
referent  of  the  pronoun  “he”,  the  system  needs  to  use  knowledge  of  reasonable  or  typical 
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behavior  for  the  two  types  of  people  mentioned  (terrorist  group  leaders  and  mayors).  The 
system  should,  of  course,  select  the  FMLN  leader  as  the  appropriate  referent  in  this  example. 

The  FMLN  leader  kidnapped  the  Andahua  mayor  and 
[he]  destroyed  the  government  house. 

The  following  example  contrasts  with  the  one  above  in  that  the  anaphoric  pronoun  refers  to 
the  object  of  the  first  sentence  instead  of  the  subject.  The  sentence  structure  in  this  example 
is  very  similar  to  that  of  the  sentence  above.  The  NLP  system  must  use  general  knowledge 
of  woundings  to  decide  which  candidate  referent  would  be  the  most  likely  interpretation  of 
the  pronoun  “he”. 

The  FMLN  leader  kidnapped  the  wounded  mayor  on  January  1st 
and  [he]  subsequently  died. 

In  the  following  example,  there  are  two  candidate  referents  for  the  pronoun  “they”  (in  brack¬ 
ets),  the  Shining  Path  guerrillas  and  the  three  government  officials.  To  decide  which  is  the 
logical  choice  to  be  the  referent  of  the  pronoun  ‘ihey”,  the  system  needs  to  use  general 
knowledge  about  the  act  of  shooting,  the  participants,  and  possible  results  of  such  an  act. 
The  system  should,  of  course,  select  the  government  officials  as  the  appropriate  referent  of 
“they”  in  this  example.  Example: 

Shining  Path  guerrillas  shot  three  government  officials  in  Andahua. 
[They]  died  in  the  hospital. 

Criteria:  Correctly  determined  which  entity,  previously  mentioned  in  the  same  or  preceding 
sentence,  is  refered  to  by  the  ambiguous  anaphoric  reference. 


Score: 


1.1. 1.2  Anaphoric  References  with  Verb  Phrase  or  Sentence  Antecedents 

This  subsection  tests  a  system’s  understanding  of  an  anaphoric  reference  (pronoun)  used  to 
refer  back  to  an  event,  activity,  or  happening  that  was  previously  expressed  using  a  verb 
phrase  or  sentence.  The  pronouns  that  are  commonly  used  to  refer  to  events,  activities, 
or  happenings  are  “this”,  “ that ”,  “these”,  “those”,  and  “it”.  Be  sure  that  the  anaphoric 
reference  that  you  create  is  NOT  ambiguous. 

In  the  following  example,  the  pronoun  “this”  refers  to  John  Smith’s  earning  over  $40,000  in 
1989.  So  the  referent  of  “this”  is  the  same  as  the  action  expressed  by  the  verb  phrase  “earn 
more  than  $1 0,000  in  1989”. 

[DO-Verb]  [NP]  [Antecedent  VP]? 

[DO-Verb]  [NP]  [DO-Verb]  [NP  +pro]? 
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Eg,  Did  John  Smith  [earn  more  than  $40,000  in  1989]? 

Did  Mary  Jones  do  [this]  ? 

In  the  following  example,  the  pronoun  “that”  in  the  second  sentence  refers  to  the  event 
expressed  by  the  first  sentence,  namely,  John  Smith’s  being  hired.  Thus  the  entire  first 
sentence  is  the  antecedent  expression  for  the  pronoun  “that”. 

[Antecedent  S]? 

[DO- Verb]  [NP  +pro]  [VP]? 

Eg,  [When  was  John  Smith  hired]  ? 

Did  [that]  bring  the  number  of  employees  to  500  ? 

In  the  following  example,  the  pronoun  “this”  in  the  second  sentence  refers  to  the  event 
expressed  by  the  first  sentence,  namely,  the  explosion  of  the  Avianca  727  on  November  7th. 
Thus  the  entire  first  sentence  is  the  antecedent  expression  for  the  pronoun  “this”. 

[Antecedent  S]. 

[NP]  [Verb]  [NP  +pro]  [NP]  ([Complement]). 

Eg,  [On  November  7,  an  Avianca  727  exploded  in  midair  near  Bogota.] 
Authorities  attribute  [this]  to  the  Sinister  Extraditables. 

When  [Antecedent  S],  [NP  +pro]  [Verb  -{-passive]  to  [VP], 

Eg,  When  [the  Avianca  727  was  destroyed  near  Bogota] , 

[it]  was  believed  to  be  a  terrorist  bombing  incident. 

Criteria:  Correctly  determined  which  event,  activity,  or  happening  (one  or  more)  previ¬ 
ously  mentioned  in  the  same  or  preceding  sentence,  is  refered  to  by  the  anaphoric  reference 
(pronoun). 

_ _ _ _____ _ Score: _ 


1.1.2  Nominal  Anaphora 

In  addition  to  pronouns,  definite  noun  phrases  (i.e.,  a  noun  phrase  containing  the  definite 
article  “the”)  can  function  as  anaphoric  referring  expressions.  Section  1.1. 2.1  addresses  the 
case  in  which  the  antecedent  expression  is  also  a  noun  phrase,  while  Section  1.1. 2. 2  addresses 
the  case  in  which  the  antecedent  expression  is  a  clause  or  sentence.  In  the  following  sections, 
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the  symbol  [Anaphoric  NP]  represents  a  definite  noun  phrase  (i.e.,  a  noun  phrase  including 
the  definite  article  “the”). 

1.1. 2.1  Anaphoric  References  with  NP  Antecedents 

Anaphoric  references  consisting  of  definite  noun  phrases  are  typically  classified  into  two 
types:  direct  and  indirect.  These  two  types  are  covered  and  defined  in  the  following  two 
subsections. 

1.1. 2. 1.1  Direct 

A  definite  noun  phrase  receives  direct  anaphoric  interpretation  when  the  same  noun  head 
has  already  occurred  in  the  text,  and  it  is  clear  that  a  relation  of  coreference  exists  between 
the  two  noun  phrases  (Quirk  et.  al.,  1985,  p.  267).  “Coreference”  means  that  the  two  noun 
phrases  refer  to  the  same  entity.  In  this  subsection,  since  we  are  testing  the  direct  type 
of  anaphoric  reference,  be  sure  that  the  same  noun  head  is  used  in  the  anaphoric  refering 
expression  and  in  the  preceeding  corefering  expression. 

In  the  following  example,  “the  benefits”  in  the  second  sentence  is  an  anaphoric  reference  to 
the  employee  benefits  mentioned  in  the  first  sentence.  Note  that  the  same  noun  head  is  used 
in  both  noun  phrases,  namely,  “benefits”. 

[BE- Verb]  [Antecedent  NP]  [VP]? 

[DO- Verb]  [Anaphoric  NP]  [VP]? 

Eg,  Are  [employee  benefits]  listed  in  the  personnel  records  ? 

Do  [the  benefits]  include  a  pension  plan  ? 

In  the  following  example,  “these  managers”  in  the  second  sentence  is  an  anaphoric  reference 
to  the  managers  with  Ph.Ds  mentioned  in  the  first  sentence.  Note  that  the  same  noun  head 
is  used  in  both  noun  phrases,  namely,  “managers”. 

List  [Antecedent  NP]. 

[DO- Verb]  [Anaphoric  NP]  [VP]? 

Eg,  List  [the  managers  with  Ph.Ds]. 

Do  [these  managers]  earn  over  $5,000  a  month  ? 

In  the  following  example,  “the  bomb”  in  the  second  sentence  is  an  anaphoric  reference  to  the 
car  bomb  mentioned  in  the  first  sentence.  Note  that  the  same  noun  head  is  used  in  both 
noun  phrases,  namely,  “bomb”. 

([AdvP])  [Antecedent  NP]  [VP]. 

[Anaphoric  NP]  [VP]. 
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Eg,  On  January  5,  [a  car  bomb]  exploded  in  front  of  the  U.S.  embassy. 
[The  bomb]  broke  windows,  but  no  injuries  were  reported. 


Criteria:  Correctly  determined  which  object(s)  or  event(s),  previously  mentioned  in  the  same 
or  preceding  sentence,  is  refered  to  by  the  anaphoric  reference  (definite  noun  phrase). 


Score: 


1.1. 2. 1.2  Indirect 

Indirect  anaphora  occurs  when  a  reference  is  made  to  an  object  or  entity  that  has  become  part 
of  the  reader’s  (or  hearer’s)  knowledge  indirectly,  not  by  direct  mention,  but  by  association 
or  inference  from  what  has  already  been  mentioned  (Quirk  et.  al.,  1985,  p.  267).  The 
anaphoric  reference  is  made  with  a  definite  noun  phrase.  For  example,  once  a  building 
is  mentioned,  then  its  parts  are  within  the  focus  of  the  discourse  context  without  being 
explicitly  mentioned  and  can  be  refered  to  with  a  definite  noun  phrase.  In  the  first  example 
below,  after  the  mention  of  the  government  house,  any  part  thereof  can  be  refered  to  with  a 
definite  noun  phrase  such  as  “ the  left  wing”. 

[NP]  [Verb]  [Antecedent  NP]. 

[Anaphoric  NP]  [VP]. 

Eg,  FMLN  terrorists  bombed  [the  Andahua  government  house] . 

[The  left  wing]  was  destroyed. 

In  the  next  example  below,  after  the  mention  of  the  particular  massacre,  any  aspect  or  part 
of  the  massacre  can  be  refered  to  with  a  definite  noun  phrase  since  it  is  automatically  within 
the  discourse  focus  by  association  with  the  massacre.  So  in  this  example,  the  noun  phrase 
“ these  murders”  is  understood  to  refer  to  murders  that  were  part  of  the  massacre  mentioned 
previously. 

[Antecedent  NP]  [VP]. 

[NP]  [BE- Verb]  [Adjective]  [Prep]  [Anaphoric  NP]. 

Eg,  [The  massacre  of  the  Jesuit  priests  and  their  maids  in  El  Salvador] 
caused  an  international  outcry  against  the  Salvadoran  government. 
Members  of  the  Salvadoran  armed  forces  were  responsible  for 
[these  murders] . 

You  are  to  enter  input  to  the  NLP  system  that  consists  of  one  or  more  sentences  in  which 
an  object  or  event  is  mentioned  and  subsequently  a  reference  to  some  aspect  or  part  of  the 
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object  or  event  is  mentioned  using  a  definite  noun  phrase.  The  latter  definite  noun  phrase 
should  NOT  explicitly  mention  the  original  object  or  event. 

Criteria:  Correctly  determined  the  referent  (meaning)  for  the  anaphoric  definite  noun  phrase. 
This  referent  must  be  the  correct  aspect  or  part  of  the  object  or  event  which  was  previously 
mentioned  in  the  same  or  preceeding  sentences. 


Score: 


1.1. 2.2  Anaphoric  References  With  Clause  or  Sentence  Antecedents 

This  subsection  tests  a  system’s  understanding  of  an  anaphoric  definite  noun  phrase,  used 
to  refer  back  to  an  event,  activity,  or  happening  that  was  previously  expressed  using  a  clause 
or  sentence.  When  entering  input  to  the  NLP  system,  be  sure  that  the  anaphoric  reference 
that  you  use  is  NOT  ambiguous. 

In  the  following  example,  the  definite  noun  phrase  “the  incident  ”  is  an  anaphoric  expression 
that  refers  to  the  event  described  by  the  first  sentence. 

[Antecedent  Sj. 

[Anaphoric  NP]  [VP]. 


Eg,  [On  January  5,  a  car  bomb  exploded  in  front  of  the  U.S.  embassy]. 

[The  incident]  resulted  in  some  damage,  but  no  injuries  were  reported. 


In  the  following  example,  the  definite  noun  phrase  “the  incident”  in  the  second  sentence  is 
an  anaphoric  expression  that  refers  to  the  event  described  by  the  first  sentence. 

[Antecedent  S]. 

[NP]  [Verb]  [NP]  [Prep]  [Anaphoric  NP]. 

Eg,  [On  January  5,  a  car  bomb  exploded  in  front  of  the  U.  S.  embassy]. 
The  ELN  claimed  responsibility  for  [the  incident] . 


In  the  following  example,  the  definite  noun  phrase  “ the  event”  is  an  anaphoric  expression 
that  refers  to  the  happening  described  by  the  first  sentence. 

When  [Antecedent  S],  [Anaphoric  NP]  [Verb  +passive]  to  [VP]. 


Eg,  [When  the  Avianca  727  was  destroyed  near  Bogota] , 
[the  event]  was  believed  to  be  a  terrorist  bombing. 
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Criteria:  Correctly  determined  which  event,  activity,  or  happening,  previously  expressed  by 
a  clause  or  sentence,  is  refered  to  by  the  anaphoric  reference  (definite  noun  phrase). 

— - „  Score: _ 

1.1.3  Anaphora  with  SO  and  A  5 

Other  anaphoric  devices  include  the  pro-predicates  “so”  and  “as”. 

The  Human  Rights  Commission  of  the  National  Dialogue  expressed  concern  over 
the  violence,  and  [so]  did  the  highest  government  officials. 

The  Human  Rights  Commission  of  the  National  Dialogue  expressed  concern  over 
the  violence,  [as]  did  the  highest  government  officials. 

In  the  above  examples,  the  expressions  “so”  and  “as”  are  anaphoric  expressions  that  have 
the  verb  phrase  “expressed  concern  over  the  violence”  as  antecedent  expression.  The  latter 
clause  in  each  of  the  above  sentences  could  be  expressed  as: 

The  highest  government  officials  did  [express  concern  over  the  violence]. 

In  the  following  patterns,  the  [Aux-Verb]  is  filled  by  a  closed  class  of  verbs  which  includes 
the  classes  described  as  “Primary”  and  “Modal”  in  Sections  I  and  II  of  the  Procedure.  The 
[Aux-Verb]  of  the  anaphoric  clause  is  the  same  as  that  of  the  antecedent  sentence,  except 
in  the  case  of  open  class  verbs  in  the  present  tense.  If  the  verb  in  the  antecedent  sentence 
is  an  open  class  verb  in  the  present  tense,  the  [Aux-Verb]  in  the  clause  beginning  with  “so” 
or  “as”  is  filled  by  “do”.  The  main  verb  is  omitted  in  the  anaphoric  clause  unless  it  is  a 
Primary  Verb  (i.e.,  “be”,  “do”,  “have”). 

In  the  following  patterns  and  examples,  the  conjunction  “and”  could  be  eliminated  and  the 
two  embedded  sentences  could  each  be  stated  separately.  For  alternative  variation,  a  comma 
could  be  inserted  at  the  end  of  the  first  embedded  sentence. 

Pattern:  [Aux-Verb]  [NP]  [Verb]  ([NP])  ([Adverbial])? 

Example:  Did  John  Smith  earn  $30,000  in  1989  ? 

([Aux-Verb])  [NP]  [DO- Verb]  so  (also)? 

Did  Mary  Harris  do  so  also? 

John  Smith  did  so  also? 

Pattern:  [NP]  [Verb]  ([NP])  and  so 

Example:  The  ELN  kidnapped  three  government  officials  and  so 
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([Aux-Verb])  [DO-Verb]  [NP]. 

did  the  FARC. 

Pattern:  [NP]  [Verb]  ([NP])  as  ([Aux-Verb])  [DO-Verb]  [NP]. 

Pattern:  [NP]  [Verb]  ([NP])  and 

Example:  The  ELN  kidnapped  several  government  officials  and 

[NP]  ([Aux-Verb])  [DO-Verb]  so  (also), 

the  FARC  did  so  also. 


1.1. 3.1  Anaphora  with  SO 

In  this  subsection,  test  the  use  of  “ so ”  as  an  anaphoric  reference  which  refers  back  to  a 
situation,  event,  or  predicate  expressed  by  a  verb  phrase  or  sentence. 

In  the  first  example  below,  the  anaphoric  pronoun-like  word  “so”  refers  to  the  attribute  or 
predicate  “earns  $30,000”  expressed  as  part  of  the  first  sentence. 

Eg,  John  Smith  [earns  $30,000],  doesn't  he? 

Mary  Harris  does  [so]  also,  doesn’t  she? 

Eg,  Did  John  Smith  [earn  over  $30,000  in  sales  commissions]? 

Did  Mary  Harris  do  [so]  also? 

Eg,  The  ELN  [kidnapped  several  government  officials]  in 
????  and  the  FARC  did  [so]  in  ????? 

Eg,  The  ELN  has  [claimed  responsibility  for  the  bombing 
of  the  government  house] ,  and  [so]  has  the  FARC . 

Criteria:  Correctly  determined  which  action,  activity,  predicate,  or  happening,  previously 
expressed  by  a  verb  phrase  or  sentence  in  the  same  or  preceding  sentence,  is  refered  to  by 
the  anaphoric  pro- word  “so”. 


Score: 


1.1. 3. 2  Anaphora  with  AS 

In  this  subsection,  test  the  use  of  “as”  as  an  anaphoric  reference  which  refers  back  to  a 
situation,  event,  or  predicate  expressed  by  a  verb  phrase  or  sentence. 

In  the  first  example  below,  the  anaphoric  pronoun-like  word  “as”  refers  to  the  attribute  or 
predicate  “earns  $30,000”  expressed  as  part  of  the  first  sentence.  The  clause  “as  does  Mary 
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Harris”  is  used  as  a  presupposition  and  if  you  use  a  test  input  such  as  this,  you  must  make 
sure  that  the  presupposition  is  true. 

Eg,  Does  John  Smith  [earn  over  $30,000] 

[as]  does  Mary  Harris? 

Eg,  Did  John  Smith  [earn  over  $30,000  in  commissions  in  1989] 

[as]  did  Mary  Harris? 

Eg,  The  ELN  has  [claimed  responsibility  for  the  bombing] , 

[as]  has  the  FARC. 

Criteria:  Correctly  determined  which  action,  activity,  predicate,  or  happening,  previously 
expressed  by  a  verb  phrase  or  sentence  in  the  same  or  preceding  sentence,  is  refered  to  by 
the  anaphoric  pro- word  “as”. 


Score: 


1.1.4  Intra-  and  Inter- Sentential  Anaphora 

The  expression  which  provides  the  referent  of  an  anaphoric  expression  must  precede  the 
anaphoric  expression  in  the  discourse.  However,  it  may  be  a  constituent  of  the  same  sentence, 
called  intra-sentential  anaphora,  or  it  may  be  a  constituent  of  some  preceding  sentence, 
called  inter-sentential  anaphora.  The  preceding  sections  on  anaphora  have  included  both. 
We  include  an  example  of  each  here.  Test  the  system  for  each  type. 

1.1. 4.1  Intra-sentential  anaphora 

Test  the  NLP  system’s  ability  to  understand  intra-sentential  anaphora. 

Eg,  Did  [Mary  Harris]  meet  [her]  sales  quota  for  July  ? 

Eg,  The  Shining  Path  guerrillas  murdered  [the  mayor]  by  bombing 
[his]  home. 

Criteria:  Correctly  determined  which  entity,  previously  mentioned  in  the  same  sentence,  is 
refered  to  by  the  anaphoric  reference  (pronoun). 

_ Score:  _ 

1.1. 4. 2  Inter- sentential  anaphora 

Test  the  NLP  system’s  ability  to  understand  inter-sentential  anaphora. 

Eg,  Which  [employees]  earn  over  $5,000  a  month  ? 

Are  [they]  salespersons  ? 

Eg,  Shining  Path  guerrillas  kidnapped  [the  mayor]  on  January  4. 

The  guerrillas  subsequently  killed  [him] . 
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Criteria:  Correctly  determined  which  entity,  previously  mentioned  in  a  preceding  sentence, 
is  refered  to  by  the  anaphoric  reference  (pronoun). 


Score: 


1.2  Cataphoric  Reference 

Cataphoric  reference  is  the  opposite  of  anaphoric  reference  in  that  in  the  case  of  a  cataphoric 
refering  expression,  the  expression  which  provides  the  unique  referent  follows  the  cataphoric 
expression  in  the  discourse,  instead  of  preceding  it  as  in  the  case  of  anaphora.  Thus  a 
cataphoric  reference  is  actually  an  anticipatory  reference,  where  the  item  referred  to  follows 
the  expression  that  refers  to  it  initially.  Cataphoric  reference  is  used  less  frequently  than 
anaphoric  references.  In  the  following  example,  the  pronoun  “it”  is  a  cataphoric  expression 
that  refers  to  the  the  Human  Rights  Commission  of  the  National  Dialog  mentioned  later  in 
the  discourse. 

After  [it]  emerged  from  the  Esquipulas  Peace  Agreement,  [the  Human  Rights  Com¬ 
mission  of  the  National  Dialog]  expressed  concern  over  the  current  escalation  of 
violence  in  Guatemala. 

Anticipatory  references  are  called  “cataphora”,  since  the  ’ana-’  prefix  in  “anaphora”  is  de¬ 
rived  from  Greek,  and  means  “upward,  backward”  indicating  the  direction  of  reference  to  a 
previous  item  called  an  “antecedent”  (in  turn  derived  from  Latin  ’ante-’  and  ’cedere’,  mean¬ 
ing  “to  go  before”),  whereas  ’cata-’  has  the  sense  “down,  downward”,  indicating  a  referent 
further  down  in  the  discourse,  which  we  will  call  the  “postcedent”. 


1.2.1  Cataphoric  References  With  NP  Postcedents 

In  this  subsection,  we  test  a  system’s  ability  to  determine  the  meaning  of  cataphoric  ref¬ 
erences  that  have  referents  expressed  as  noun  phrases.  The  referent  would  typically  be 
an  object  or  entity.  In  the  first  example  below,  the  pronoun  “he”  is  used  as  a  cataphoric 
reference  to  the  person  (entity)  John  Smith  refered  to  subsequently  in  the  same  sentence. 

[Conj  -Psubord]  [NP  +pro]  [VP],  [DO-Verb]  [Postcedent  NP]  [VP]? 

Eg,  Before  [he]  was  promoted,  did  [John  Smith]  earn  over  $30,000? 

Eg,  When  [he]  was  promoted,  did  [John  Smith]  work  in  the  sales  department? 

[Conj  +subord]  [NP  Tpro]  [VP],  [Postcedent  NP]  [VP] 


Eg,  When  [he]  was  in  Bogota  writing  an  article  about  the  drug  war, 
[the  Brazilian  journalist  Juan  Castellar]  was  kidnapped  by 
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Colombian  Army  of  National  Liberation  guerrillas. 

Eg,  When  [he]  was  kidnapped  by  Colombian  Army  of  National  Liberation 
guerrillas,  [the  Brazilian  journalist  Juan  Castellar]  was  in 
Bogota  writing  an  article  about  the  drug  war. 

(Conj  +subord]  [NP  4-pro]  [VP],  [Postcedent  NP]  [VP]. 

Eg,  Before  [they]  appeared  in  Andahua,  [the  Shining  Path  guerrillas] 
murdered  the  mayor. 

[Conj  +subord]  [VP]  [NP  +pro],  [Postcedent  NP]  [VP] 

Eg,  When  Colombian  Army  of  National  Liberation  guerrillas 

kidnapped  [him] ,  [the  Brazilian  journalist  Juan  Castellar] 
was  in  Bogota  writing  an  article  about  the  drug  war. 

[Conj  +subord]  [NP]  [Verb]  [NP  4-pro]  ([Complement]),  [Postcedent  NP]  [VP], 

Eg,  When  the  authorities  accused  [them]  of  murdering  the  mayor, 

[the  Shining  Path  guerrillas]  were  in  the  town  of  Andahua. 

Criteria:  Correctly  determined  which  entity,  subsequently  mentioned  using  a  noun  phrase  in 
the  same  or  a  following  sentence,  is  refered  to  by  the  cataphoric  reference  (pronoun). 


Score: 


1.2.2  Cataphoric  References  With  Verb  Phrase  or  Sentence  Antecedents 

In  this  subsection,  we  test  a  system’s  ability  to  determine  the  meaning  of  cataphoric  refer¬ 
ences  that  have  referents  (meanings)  expressed  as  verb  phrases  or  sentences.  The  referent 
(meaning)  would  typically  be  an  event,  happening,  or  state  of  affairs.  In  the  first  example 
below,  the  pronoun  “it"  is  used  as  a  cataphoric  reference  to  the  event  of  John  Smith’s  earn¬ 
ing  over  $40,000  in  1989,  expressed  subsequently  as  an  embedded  sentence  within  the  same 
sentence. 


Eg,  Is  [it]  true  that  [John  Smith  earned  over  $40,000  in  1989]? 
Eg,  When  was  [it]  that  [John  Smith  was  hired]? 

Eg,  Was  [it]  in  1989  that  [John  Smith  was  hired]? 
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Eg,  Is  [it]  the  case  that  [John  Smith  earns  over  $40,000]? 

Eg,  [It]  was  announced  that  [the  FMLN  had  kidnapped  the  mayor] . 

Eg,  After  [it]  was  announced  that  [the  FMLN  had  bombed  the 

governor's  car],  the  FMLN  also  bombed  the  government  house. 

Eg,  When  [it]  was  announced  that  [the  Andahua  mayor  had 
been  kidnapped] ,  the  FMLN  announced  ransom  demands . 

Eg,  When  authorities  announced  [it] ,  the  FMLN  denied  [having 
bombed  the  government  house] . 

Eg,  After  [it]  was  discovered  that  [the  FMLN  had  kidnapped  the  mayor] , 
police  found  his  body  in  the  woods. 

Eg,  After  [it]  was  confirmed  by  the  FMLN  leader,  a  government 

spokesman  announced  that  [the  mayor  had  been  kidnapped  by  the  FMLN] . 

Criteria:  Correctly  determined  which  event,  happening,  or  state  of  affairs,  subsequently 
mentioned  using  a  verb  phrase  or  sentence  in  the  same  or  a  following  sentence,  is  refered  to 
by  the  cataphoric  reference  (pronoun). 


Score: 


1.3  General  Knowledge  or  the  Larger  Situation 

The  identity  of  the  referent  of  a  phrase  may  be  evident  from  knowledge  of  the  ‘larger  situation’ 
which  the  writer  (speaker)  and  reader  (hearer)  share.  The  ‘larger  situation’  may  be  some 
application  domain  or  may  be  so  encompassing  as  to  be  general  world  knowledge.  (Quirk 
et.  al.,  1985,  p.  266) 

In  this  subsection,  we  will  treat  the  application  domain  of  the  NLP  system  or  some  more 
limited  context  established  by  input  text  as  the  ‘larger  situation’.  You  are  to  test  the  system’s 
understanding  of  some  definite  noun  phrases  which  the  system  should  be  able  to  interpret 
based  on  knowledge  of  the  domain. 

In  the  first  examples,  the  referent  of  the  definite  noun  phrase  “the  president”  should  be  the 
president  of  Xidec  for  the  Xidec  employee  database  application. 

Eg,  What  is  the  salary  of  [the  president]? 

Eg,  Who  is  [the  president]? 

In  the  next  example,  the  referent  of  the  definite  noun  phrase  “the  mayor”  should  be  the 
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mayor  of  Andahua,  a  government  official. 

Eg,  When  the  Andahua  bank  exploded,  [the  mayor]  was  shot 
by  the  FMLN  as  he  rushed  into  the  street. 

Criteria:  Correctly  determined  the  referent  (meaning)  for  the  situational  reference  (a  definite 
noun  phrase).  This  referent  is  the  correct  object  or  entity,  which  is  known  to  the  system 
because  of  its  knowledge  of  the  larger  situation  or  application  domain. 


Score: 


2.  Generic  Reference  (t.b.d.) 

In  the  sentence  “Tigers  are  dangerous  animals,  the  reference  “ Tigers ”  is  a  generic  reference, 
since  we  are  thinking  of  the  class  ‘tiger’  without  specific  reference  to  particular  tigers.  Sin¬ 
gular  or  plural,  definite  or  indefinite,  noun  phrases  can  often  be  used  without  appreciable 
difference  of  meaning  in  generic  contexts,  as  in  the  following  examples.  (Quirk  et.  al.,  1985, 
p.265) 

[A  tiger]  is  a  dangerous  animal. 

[Tigers]  are  dangerous  animals. 

] The  tiger]  can  be  a  dangerous  animal. 

In  other  words,  we  are  testing  whether  an  NLP  system  can  treat  any  class  or  type  as  such 
(namely,  as  a  class  or  type),  rather  than  as  a  collection  of  individuals.  That  is,  does  the 
system  know  anything  about  the  class  as  a  whole,  rather  than  just  information  about  the 
individuals  within  the  class  or  group? 

You  are  to  test  the  ability  of  the  NLP  system  to  understand  generic  references.  Submit 
input  to  the  system  which  includes  a  generic  noun  phrase.  Use  any  of  the  three  types  of 
noun  phrases  listed  above. 

In  the  first  example,  for  the  employee  database  domain,  we  inquire  whether  the  system  knows 
whether  the  Xidec  employees,  as  a  group,  are  well  paid. 

Eg,  Are  Xidec  employees  well  paid? 

For  the  terrorist  domain,  we  test  whether  the  system  can  treat  the  concept  of  ‘terrorists’  as 
a  class  or  type. 

Eg,  [Terrorists]  were  responsible  for  over  50  kidnappings 
in  South  America  during  1989. 
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Criteria:  Correctly  determined  the  referent  (meaning)  for  the  generic  reference  (a  noun 
phrase).  The  referent  must  be  a  general  class  or  type  of  entity,  not  a  specific  individual  or 
group  of  individuals  of  the  mentioned  type. 


XI.  ELLIPSIS 


Ellipsis  may  be  defined  as  the  omission  of  grammatical  elements  of  a  sentence  which  are 
predictable  and  recoverable  from  context.  In  some  special  cases  (e.g.,  comparative  clauses, 
coordinate  constructions)  the  possibilities  of  omission  are  closely  determined  by  the  rela¬ 
tion  between  the  antecedent  construction  and  the  elliptical  construction  within  the  larger 
construction  of  which  they  are  both  a  part.  The  special  ellipsis  types  are  covered  in  the 
Procedure  sections  on  the  respective  construction  types  (e.g.,  comparative  constructions, 
coordinate  constructions).  This  section  of  the  Procedure  focuses  on  non-special  (general) 
ellipsis  in  which  the  relation  between  the  two  constructions  does  not  determine  or  restrict 
the  possibilities  for  omission. 

According  to  Quirk  et.  al.  (1985),  the  types  of  recoverability  include: 

•  Textual:  the  full  grammatical  element  can  be  recovered  from  a  neighboring  part  of  the 
text  (e.g.,  “Jane  Smith  earns  more  money  than  Sue  Jones  (earns).”) 

•  Situational:  the  full  grammatical  element  can  be  recovered  from  the  extralinguistic 
situation  (e.g.,  (“He/She)  Doesn’t  look  too  well”). 

•  Structural:  the  full  grammatical  element  can  be  recovered  through  knowledge  of  the 

grammatical  structure.  For  exa»cVie,  in  the  sentence  “It  is  true  (that)  George  Bush 
is  president  of  the  U.S.”,  the  coi  “that”  is  recoverable  from  knowledge  of  the 

structure  of  such  a  sentence  with  a;>  on. bedded  clause. 

Many  forms  of  structural  ellipsis  are  covered  in  the  different  Procedure  sections  that  focus  on 
the  different  types  of  grammatical  structures,  without  specifically  referring  to  the  particular 
phenomenon  as  ellipsis.  For  example,  a  certain  form  of  structural  ellipsis  in  relative  clauses 
produces  what  are  referred  to  as  reduced  relative  clauses.  However,  in  this  Application 
Procedure,  the  concept  of  reduced  relative  clauses  is  covered  in  the  section  on  Noun  Phrases 
without  explicitly  identifying  them  as  a  form  of  structural  ellipsis. 

General  textual  ellipsis  is  also  called  standard  ellipsis  and  is  the  focus  of  this  section  of  the 
Procedure.  In  this  type  of  ellipsis,  the  elliptical  element  can  be  recovered  from  neighboring 
text,  it  is  typically  the  final  element  of  a  construction,  and  it  is  typically  anaphoric  (as 
opposed  to  cataphoric).  Anaphoric  means  that  the  recovery  of  the  missing  grammatical 
element  can  be  ascertained  from  earlier  or  preceding  text.  Typically,  a  grammatical  pattern 
(e.g.,  noun  phrase,  verb  phrase,  clause)  is  used  twice  in  succession:  the  first  time  with 
complete  constituents,  the  second  time  with  omitted  elements  that  can  be  recovered  by 
comparing  the  language  structure  to  the  similar  preceding  structure. 

In  the  examples  of  this  section,  ellipted  (omitted)  sentence  elements  are  included  in  paren¬ 
theses,  but  are  not  to  be  used  as  part  of  the  test  input  to  an  NLP  system. 
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1.  Elliptical  noun  phrases 

The  elements  ellipted  in  noun  phrases  are  most  frequently  the  final  elements  of  the  phrases. 
This  means  that  the  ellipsis  tends  to  follow  the  right-to-left  precedence  shown  below: 


[Determinative] 


[Premodification] 


[Head] 


[Postmodification] 


For  noun  phrase  ellipsis,  a  noun  phrase  pattern  is  used  twice.  The  second  time  it  is  used, 
one  or  more  phrase  elements  are  omitted,  but  can  be  recovered  from  the  closely  preceding 
use  of  the  same  pattern. 

1.1  Ellipsis  of  head  only,  no  postmodifiers  present 

For  this  case  of  ellipsis,  when  the  noun  phrase  pattern  is  used  the  first  time,  the  noun  phrase 
must  include  a  head  element  and  usually  includes  a  determinative  and  premodifier.  When 
used  the  second  time,  the  head  alone  is  omitted.  Remember  that  the  words  in  parentheses 
are  the  omitted  words  and  are  not  included  in  the  text  submitted  to  the  NLP  system.  Thus 
using  first  sentential  example  below,  the  evaluator  would  first  input  “Who  is  the  youngest 
employee?”  to  the  system,  wait  for  a  correct  response,  and  then  input  “Who  is  the  oldest?” 
without  the  word  “employee”  and  determine  whether  the  system  understood  correctly. 

Noun  Phrase  Pattern  and  Examples: 

[Determinative]  [Premodification]  [Head] 

1st:  the  youngest  employee 

2nd:  the  oldest  (employee) 

Sentential  Examples: 

Eg,  Who  is  the  youngest  employee? 

Who  is  the  oldest  (employee)? 

Eg,  The  terrorist  attack  on  Andahua  killed  the  oldest  villager. 

The  terrorist  attack  on  Andahua  also  killed  the  youngest 
(villager) . 

Eg,  Several  civilians  were  killed  and  several  (civilians) 
were  wounded  by  the  ELN  attack. 

Eg,  Seven  soldiers  were  killed  and  several  (soldiers)  were 
wounded  by  a  bomb  when  a  group  of  ELN  rebels  ambushed  a 
military  convoy  in  Saxavena. 

Eg,  The  guerrillas  kidnapped  the  richest  businessman  and  the 
most  famous  (businessman) . 

Eg,  One  pe  sant  was  killed  and  another  (peasant)  was  wounded 
during  the  guerrilla  attack  on  the  presidential  farm. 

Criteria:  Demonstrated  that  it  determined  the  correct  head  for  the  noun  phrase  whose  head 
had  been  omitted,  and  correctly  interpreted  the  noun  phrase. 


Score: 


1.2  Ellipsis  of  premodifier(s)  and  head,  no  postmodifiers  present 

For  this  case  of  ellipsis,  when  the  noun  phrase  pattern  is  used  the  first  time,  the  noun  phrase 
must  include  a  head  element  and  a  premodifier  and  usually  includes  a  determinative.  When 
used  the  second  time,  the  head  and  last  (right-most)  premodifier(s)  are  omitted. 

Noun  Phrase  Pattern  and  Examples: 

[Determinative]  [Premodification]  [Head] 

1st:  the  oldest  New  York  employee 

2nd:  the  youngest  (New  York)  (employee) 

Sentential  Examples: 

Eg,  Who  is  the  oldest  New  York  employee? 

Who  is  the  youngest  (New  York  employee)? 

Eg,  The  ambush  on  February  1st  was  the  most  unexpected  terrorist 
action.  The  ambush  was  also  the  least  organized  (terrorist 
action) . 

Eg,  The  kidnapping  on  February  1st  was  the  first  terrorist 
action.  The  assassination  on  March  4th  was  the  second 
(terrorist  action) . 

Eg,  The  guerrillas  kidnapped  the  richest  Guatemalan  businessman 
and  the  most  famous  (Guatemalan  businessman) . 

Eg,  A  young  peasant  was  killed  and  another  (young  peasant)  was 
wounded  in  a  guerrilla  attack. 

Criteria:  Demonstrated  that  it  determined  the  correct  premodifier  and  head  for  the  noun 
phrase  whose  premodifier  and  head  had  been  omitted,  and  correctly  interpreted  the  noun 
phrase. 

_ Score:  _ 


1.3  Ellipsis  of  post-modifier(s)  only 

For  this  case  of  ellipsis,  when  the  noun  phrase  pattern  is  used  the  first  time,  the  noun  phrase 
must  include  a  head  element  and  at  least  one  postmodifier.  When  used  the  second  time,  the 
postmodifiers  are  omitted. 

Noun  Phrase  Pattern  and  Examples: 

[Determinative]  [Head]  [Postmodification] 

1st:  the  women  in  the  Computer  Science  Department 

2nd:  the  men  (in  the  Computer  Science  Department) 

Sentential  Examples: 

Eg,  List  the  women  in  the  Computer  Science  Department. 


List  the  men  (in  the  Computer  Science  Department) . 

Eg,  The  mayor  of  Andahua  was  kidnapped  on  January  8. 

The  mayor  (of  Andahua)  was  killed  on  January  10. 

Eg,  Terrorists  kidnapped  two  members  of  the  embassy  staff. 

The  terrorists  killed  the  other  members  (of  the  embassy  staff) . 
Eg,  Terrorists  burned  the  stores  in  Andahua. 

Then  the  terrorists  burned  the  banks  (in  Andahua) . 

Eg,  The  houses  on  the  presidential  farm  were  burned  and  some 

trucks  (on  the  presidential  farm)  were  stolen  by  the  guerrillas. 

Criteria:  Demonstrated  that  it  determined  the  correct  premodifier(s)  for  the  noun  phrase 
whose  premodifier(s)  had  been  omitted,  and  correctly  interpreted  the  noun  phrase. 

_ _ _ Score:  _ 


1.4  Ellipsis  of  head  and  post-modifier(s) 

For  this  case  of  ellipsis,  when  the  noun  phrase  pattern  is  used  the  first  time,  the  noun  phrase 
must  include  a  head  element  and  at  least  one  postmodifier.  When  used  the  second  time, 
both  the  head  and  postmodifiers  are  ellipted. 

Noun  Phrase  Pattern  and  Examples: 

[Determinative] [Premodification]  [Head]  [Postmodification] 

1st:  the  oldest  person  in  the  Marketing  Department 

2nd:  the  youngest  (person)  (in  the  Marketing  Department) 

Sentential  Examples: 

Eg,  Who  is  the  oldest  person  in  the  Marketing  Department? 

Who  is  the  youngest  (person  in  the  Marketing  Department)? 

Eg,  The  guerrillas  won  the  first  attack  on  Andahua. 

The  guerrillas  lost  the  second  (attack  on  Andahua) . 

Eg,  Six  terrorists  of  the  ELN  kidnapped  the  mayor. 

The  six  (terrorists  of  the  ELN)  murdered  the  governor. 

Eg,  The  first  kidnapping  of  Andahua  civilians  took  place  on 
January  9  and 

the  second  (kidnapping  of  Andahua  civilians)  took  place 
on  January  12 . 

Eg,  The  guerrillas  kidnapped  the  richest  businessman  in  Guatemala 
and  the  poorest  (businessman  in  Guatemala) . 

Criteria:  Demonstrated  that  it  determined  the  correct  head  and  postmodifier(s)  for  the  noun 
phrase  whose  head  and  postmodifier(s)  had  been  omitted,  and  correctly  interpreted  the  noun 
phrase. 

_  Score:  - 
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1.5  Ellipsis  of  premodifier(s)  and  head  and  postmodifier (s) 

For  this  case  of  ellipsis,  when  the  noun  phrase  pattern  is  used  the  first  time,  the  noun  phrase 
must  include  a  head  element,  at  least  one  premodifier,  and  at  least  one  postmodifier.  When 
used  the  second  time,  the  last  (right-most)  premodifier (s),  the  head,  and  the  postmodifiers 
are  ellipted. 

Noun  Phrase  Pattern  and  Examples: 

[Determinative]  [Premodification]  [Head]  [Postmodification] 

1st:  the  oldest  Nev  York  employee  in  sales 

2nd:  the  youngest  (Nev  York)  (employee)  (in  sales) 

Sentential  Examples: 

Eg,  Who  is  the  oldest  New  York  employee  in  sales? 

Who  is  the  youngest  (New  York  employee  in  sales)? 

Eg,  The  first  terrorist  attack  on  Usulutan  killed  30  civilians. 

The  second  (terrorist  attack  on  Usulutan)  killed  50  civilians. 
Eg,  The  June  terrorist  attack  on  Usulutan  killed  30  civilians. 

The  next  (terrorist  attack  on  Usulutan)  killed  50  civilians. 

Eg,  The  first  URNG  attack  on  the  presidential  farm  killed  a  peasant. 
The  second  (URNG  attack  on  the  presidential  farm)  caused  damage 
to  the  farm. 

Eg,  The  first  guerrilla  kidnapping  of  civilians  occurred  on  January 
10.  The  next  (guerrilla  kidnapping  of  civilians)  occurred  on 
February  2. 

Criteria:  Demonstrated  that  it  determined  the  correct  premodifier,  head,  and  postmodi- 
fier(s)  for  the  noun  phrase  from  which  these  three  elements  had  been  omitted,  and  correctly 
interpreted  the  noun  phrase. 

_ Score: _ 


1.6  Ellipsis  of  head  with  postmodifier(s)  retained  (Medial  Ellipsis) 

Medial  ellipsis  in  a  noun  phrase  occurs  when  a  postmodifier  is  present  and  retained,  while 
the  head  of  the  noun  phrase  is  ellipted.  For  this  case  of  ellipsis,  when  the  noun  phrase 
pattern  is  used  the  first  time,  the  noun  phrase  must  include  a  head  element  and  at  least  one 
postmodifier.  When  used  the  second  time,  the  head  is  ellipted,  but  the  postmodifiers  are 
retained. 

Noun  Phrase  Pattern  and  Examples: 

[Determinative]  [Premodification]  [Head]  [Postmodification] 

1st:  the  oldest  female  in  the  Marketing  Department 

2nd:  the  youngest  (female)  in  the  Marketing  Department 

Sentential  Examples: 
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Eg,  Who  is  the  oldest  female  in  the  Marketing  Department? 

Who  is  the  youngest  (female)  in  the  Marketing  Department? 

Eg,  The  first  attack  by  the  terrorists  on  Usulutan  killed 
20  civilians.  The  second  (attack)  by  the  terrorists 
on  Usulutan  killed  50  civilians. 

Eg,  The  first  attack  by  the  guerrillas  on  the  presidential  farm 
killed  a  peasant.  The  second  (attack)  by  the  guerrillas 
on  the  presidential  farm  caused  damage  to  the  farm. 

Eg,  The  first  bombing  by  the  guerrillas  caused  the  destruction 
of  the  Central  Bank.  The  next  (bombing)  by  the  guerrillas 
caused  some  damage  to  the  Government  House. 

Criteria:  Demonstrated  that  it  determined  the  correct  head  for  the  noun  phrase  whose  head 
had  been  omitted  but  not  its  premodifer(s)  or  postmodifier(s),  and  correctly  interpreted  the 
noun  phrase. 

_  Score:  _ 


2.  Elliptical  Clauses 

Ellipsis  in  clauses  is  similar  to  ellipsis  in  noun  phrases  in  that  the  ellipted  elements  tend 
to  be  the  final  (right-most)  phrase  elements.  Therefore  in  this  section,  we  start  with  the 
more  common  forms  of  ellipsis,  namely  ellipsis  of  right-most  clause  elements,  and  move  left 
in  our  consideration  of  ellipted  clause  elements.  As  in  the  case  of  noun  phrases,  the  ellipsis 
being  tested  in  the  following  subsections  will  entail  the  use  of  a  grammatical  pattern  twice 
in  succession  (within  the  same  sentence  or  in  subsequent  sentences).  Elements  are  omitted 
in  the  second  use  of  the  grammatical  pattern  such  that  they  can  be  recovered  from  the  first 
use  of  the  grammatical  pattern. 

2.1  Ellipsis  of  Subject  Complement 

Recall  that  a  subject  complement  is  an  adjective  phrase  or  a  noun  phrase  which  applies  some 
attribute  or  definition  to  the  subject,  and  it  is  generally  used  following  a  copular  verb  such 
as  “is”,  “seem”,  “become”.  The  pattern  for  the  verb  phrase  consists  of  a  verb  of  the  type 
just  mentioned,  followed  by  an  adjective  phrase  or  noun  phrase. 

Select  a  sentence  pattern  that  contains  a  subject  complement.  Use  this  pattern  to  create 
two  input  test  sentences:  the  first  sentence  should  be  complete  with  no  omitted  elements 
and  the  second  should  be  input  with  the  subject  complement  omitted  (forming  an  instance 
of  ellipsis). 

[BE-Verb]  [NP]  [Subject  Complement]  ? 

1st:  Is  John  Wilson  an  engineer  ? 

2nd:  Is  Don  Thomas  (an  engineer)  ? 

[NP]  [BE-Verb]  [Subject  Complement]  ? 
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1st : 

Which  men 

axe 

New  Yoxk  salespersons? 

2nd: 

Which  women 

are 

(New  York  salespersons) 

1st : 

The 

persons  who  killed  the  defense  minister 

are 

terrorists. 

2nd: 

The 

persons  who  killed  the  mayor 

axe 

(terrorists)  also. 

1st : 

The 

terrorist  organization  which 

conducted 

was 

the  URNG. 

an  attack  on  the  presidential 

farm 

2nd: 

The 

terrorist  organization  which 

killed 

was 

(the  URNG)  also. 

a  civilian 


Criteria:  Demonstrated  that  it  determined  the  correct  subject  complement  for  the  sentence 
from  which  the  subject  complement  had  been  omitted,  and  correctly  interpreted  the  sentence. 

_  Score:  _ 


2.2  Ellipsis  of  Subject  Complement  and  Adverbial 

For  this  type  of  ellipsis,  the  first  use  of  the  sentence  pattern  should  include  a  subject  comple¬ 
ment  and  an  adverbial.  In  the  second  use  of  the  sentence  pattern,  the  subject  complement 
and  adverbial  should  be  omitted. 


[Subject  Complement]  [Adverbial] 

Eg,  Was  John  Smith  an  employee  in  the  Sales  Department  last  year? 

Was  Paul  Martin  (an  employee  in  the  Sales  Department)  (last  year)? 

Eg,  The  kidnap  victim  was  a  cabinet  member  this  year  and 

Manuel  Baldeon 

the  murder  victim  was  (a  cabinet  member)  (this  year)  also. 

Eduardo  Lopez 

Eg,  (Preliminary  Sentence:  Guerrillas  kidnapped  a  man  and  murdered  a  man.) 

The  kidnap  victim  is  a  businessman  in  Guatemala  City. 

The  murder  victim  is  (a  businessman)  (in  Guatemala  City)  also. 

Criteria:  Demonstrated  that  it  determined  the  correct  subject  complement  and  adverbial  for 
the  sentence  or  clause  from  which  the  subject  complement  and  adverbial  had  been  omitted, 
and  correctly  interpreted  the  sentence. 

_ _ Score: _ 


2.3  Ellipsis  of  Adverbial 

For  this  type  of  ellipsis,  the  sentence  pattern  should  include  an  adverbial.  The  first  use  of 
the  sentence  pattern  should  include  the  adverbial  and  the  second  should  omit  the  adverbial. 
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[BE-Verb]  [NP]  [PP]  [Adverbial] 

1st:  Was  John  Smith  in  the  Marketing  Department  in  1989? 

2nd:  Was  Harold  Thomas  in  the  Sales  Department  (in  1989)? 

[NP]  [VP]  [Adverbial] 

1st:  Five  government  soldiers  were  killed  during  a  terrorist  attack. 

2nd:  A  military  vehicle  was  also  destroyed  (during  the  terrorist  attack). 

1st:  A  peasant  was  killed  during  the  guerrilla  attack 

on  the  presidential  farm. 

2nd:  Some  farm  facilities  were  destroyed  (during  the  guerrilla  attack 

on  the  presidential  farm)  also. 


Criteria:  Demonstrated  that  it  determined  the  correct  adverbial  for  the  sentence  from  which 
the  adverbial  had  been  omitted,  and  correctly  interpreted  the  sentence. 

_ Score: _ 


2.4  Ellipsis  of  Verb  Head 

For  this  type  of  ellipsis,  the  first  use  of  the  sentence  pattern  should  include  a  verb  phrase 
and  the  second  use  of  the  pattern  omits  the  verb  head  only. 


[BE-Verb] 

[NP]  [Verb  Head] 

[Adverbial] 

1st : 

Was 

John  Smith 

hired 

in  1979? 

2nd: 

Was 

Mike  Thomas 

(hired) 

in  1980? 

[NP] 

[VP] 

1st : 

The  FMLN  terrorists 

attacked. 

2nd:  The  Shining  Path  terrorists  will  (attack)  also. 

(Preliminary  Sentence:  Guerrillas  attacked  yesterday.) 

1st:  The  presidential  farm  was  burned. 

2nd:  A  government  building  was  also  (burned). 

Criteria:  Demonstrated  that  it  determined  the  correct  verb  head  for  the  sentence  from  which 
the  verb  head  had  been  omitted,  and  correctly  interpreted  the  sentence. 

-  Score:  _ 


2.5  Ellipsis  of  Verb  Head  and  Adverbial 

For  this  type  of  ellipsis,  the  sentence  pattern  should  include  an  adverbial  in  the  verb  phrase. 
When  the  sentence  pattern  is  used  the  first  time,  all  elements  should  be  included.  When  the 
pattern  is  used  the  second  time,  the  verb  head  and  the  adverbial  should  be  omitted. 
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[BE-Verb] 

CNP] 

[Verb] 

[Adverbial] 

1st : 

Is 

John  Smith 

working 

in  sales? 

2nd: 

Is 

Mike  Thomas 

(working) 

(in  sales)? 

1st : 

Was 

John  Smith 

hired 

in  1987? 

2nd: 

Was 

Mike  Thomas 

(hired) 

(in  1987)? 

Other  Examples: 

Eg,  How  long  has  John  Smith  worked  in  sales? 

Hov  long  has  Mike  Thomas  (worked  in  sales)? 

Eg,  The  Bogata  bank  was  bombed  in  1990  and 

the  Andahua  bank  was  (bombed  in  1990)  also. 

Eg,  The  presidential  farm  was  burned  during  the  guerrilla  attack. 

A  government  building  was  also  (burned  during  the  guerrilla  attack) . 

Criteria:  Demonstrated  that  it  determined  the  correct  verb  head  and  adverbial  for  the 
sentence  from  which  the  verb  head  and  adverbial  had  been  omitted,  and  correctly  interpreted 
the  sentence. 


Score: 


2.6  Ellipsis  of  Verb  Head  and  Object  and  Adverbial 

For  this  type  of  ellipsis,  the  sentence  pattern  should  include  a  verb  phrase  with  an  object 
and  adverbial.  When  the  sentence  pattern  is  used  the  first  time,  all  sentence  elements  are 
included.  When  the  sentence  pattern  is  used  the  second  time,  the  verb  head,  object,  and 
adverbial  are  omitted. 


[DO-Verb] 

[NP] 

[Verb] 

[Object] 

[Adverbial] 

1st:  Did 

Mary  Smith 

meet 

quota 

.last  month? 

2nd :  Did 

John  Thomas 

(meet) 

(quota) 

(last  month)? 

1st:  Did 

Mary  Smith 

earn 

a  bonus 

last  year? 

2nd:  Did 

John  Thomas 

(earn) 

(a  bonus) 

(last  year)? 

Another  example: 

Eg,  FMLN  terrorists  have  killed  over  20  citizens  in  1990  and 

the  Shining  Path  have  (killed  over  20  citizens  in  1990)  also. 

Eg,  The  URNG  guerrillas  kidnapped  two  local  officials  this  week. 

The  FMLN  terrorists  have  (kidnapped  two  local  officials  this 
week)  also. 

Criteria:  Demonstrated  that  it  determined  the  correct  verb  head,  object,  and  adverbial  for 
the  sentence  from  which  the  verb  head,  object,  and  adverbial  had  been  omitted,  and  correctly 
interpreted  the  sentence. 
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Score: 


2.7  Ellipsis  of  the  Verb  Phrase  with  Do-Support 

For  this  type  of  ellipsis,  when  the  sentence  pattern  is  used  the  second  time,  the  verb  phrase 
is  omitted  and  the  DO-verb  is  used  in  its  place. 


List  [NP]  [Relative  Clause] 

[  [Rel  Pronoun]  [VP]  ] 

1st:  List  the  salespersons  who  earn  more  than  $40,000. 

2nd:  List  the  engineers  who  do  (earn  more  than  $40,000). 

[DO-Verb]  [NP]  [VP] 

1st:  Did  John  Smith  start  at  Xidec 

2nd:  when  Mike  Thomas  did  (start  at  Xidec)? 


[NP] 

1st:  The  FMLN  terrorists 

2nd:  The  Shining  Path  terrorists 

1st:  The  10th  Army  Battalion 

2nd:  when  the  12th  Army  Battalion 

1st:  The  guerrillas 

2nd:  The  rebels 


[VP] 

attacked  yesterday . 
did  (attack  yesterday)  also. 

attacked  the  guerrillas  in  Usulutan 
did  (attack  the  guerrillas  in  Usulutan) . 

attacked  the  presidential  farm  yesterday, 
did  (attack  the  presidential  farm 
yesterday)  also. 


Criteria:  Demonstrated  that  it  determined  the  correct  verb  phrase  for  the  sentence  in  which 
a  form  of  the  DO-verb  has  been  substituted  for  the  verb  phrase,  and  correctly  interpreted 
the  sentence. 


Score: 


2.8  Medial  Verb  Head  Ellipsis 

When  an  adverbial  or  noun  phrase  follows  an  ellipted  element,  the  ellipsis  becomes  medial 
rather  than  final  (with  respect  to  the  positioning  of  the  elements  of  the  sentence). 

Select  a  sentence  pattern  with  an  object  noun  phrase  or  adverbial  following  the  main  verb. 
When  used  the  first  time,  the  sentence  should  not  have  any  omitted  elements.  When  used 
the  second  time,  omit  the  main  verb. 

Eg,  Does  Mary  Jones  earn  $30,000? 

(Does)  Ken  Smith  (earn)  $40,000? 

Eg,  The  terrorist  Commander  shot  President  Mubarto  and 
his  Lieutenant  (shot)  Vice  President  Winnego. 
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Eg,  The  guerrillas  attacked  the  presidential  farm  and  the 
rebels  (attacked)  the  Central  Bank. 

Criteria:  Demonstrated  that  it  determined  the  correct  constituent  (e.g.,  verb)  for  the  sen¬ 
tence  from  which  the  constituent  had  been  omitted  with  an  adverbial  or  noun  phrase  follow¬ 
ing  the  position  of  the  omitted  element,  and  correctly  interpreted  the  sentence. 

_  Score:  _ 


2.9  Ellipsis  of  Subject  and  Verb 

For  this  type  of  ellipsis,  both  subject  and  verb  are  omitted  during  the  second  use  of  the 
sentence  pattern. 

Eg,  Does  the  Sales  Department  employ  Mary  Davis? 

(Does  the  Sales  Department  employ)  Peter  Walker? 

Eg,  Did  John  Smith  manage  the  Data  Base  Project? 

(Did  John  Smith  manage)  The  Communications  Project? 

Eg,  Did  GE  hire  Mary  Davis? 

(Did  GE  hire)  Peter  Walker? 

Eg,  The  terrorists  invaded  Usulutan,  then  (they  invaded) 

Morazan . 

Eg,  The  terrorists  attacked  the  Government  House  with  grenades 

and  (the  terrorists  attacked)  the  embassy  building  with  bombs. 
Eg,  The  guerrillas  attacked  the  presidential  farm  and  then 
(they  attacked)  the  government  building. 

Eg,  The  guerrillas  bombed  the  American  Embassy,  then  (they  bombed) 
the  British  Embassy. 

Criteria:  Demonstrated  that  it  determined  the  correct  subject  and  verb  for  the  sentence 
from  which  the  subject  and  verb  had  been  c  nitted,  and  correctly  interpreted  the  sentence. 

- Score: _ 


2.10  Ellipsis  in  TO-Infinitive  Clauses 

A  TO-infinitive  clause  consists  of  the  word  “to”  followed  by  a  verb  phrase.  Select  a  sentence 
pattern  and  include  a  TO-infinitive  clause  for  the  first  of  two  test  sentences.  In  the  case  of 
the  second  test  sentence,  use  the  same  sentence  pattern,  but  omit  the  verb  phrase  from  the 
TO-infinitive  clause  so  that  the  clause  consists  of  only  the  word  “to”. 

Eg,  Does  John  Smith  manage  the  Computer  Department? 

Was  John  Smith  hired  to  (manage  the  Computer  Department)? 

Eg,  The  guerrillas  invaded  Usulutan  although  they 
were  warned  not  to  (invade  Usulutan) . 

Eg,  The  US  soldiers  did  not  attack  the  terrorists 
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although  they  planned  to  (attack  the  terrorists) . 

Eg,  The  guerrillas  did  not  burn  the  presidential  farm  although 
they  attempted  to  (burn  the  presidential  farm) . 

Eg,  The  guerrillas  did  not  kill  the  mayor  although  they  planned 
to  (kill  the  mayor) . 

Criteria:  Demonstrated  that  it  determined  the  verb  phrase  for  the  TO-infinitive  clause  for  the 
second  sentence  (from  which  the  verb  phrase  of  the  TO-infinitive  clause  had  been  omitted), 
and  correctly  interpreted  the  sentence. 

_  Score:  _ 


2.11  Ellipsis  of  Entire  Clauses 

Ellipsis  of  entire  clauses  is  limited  to  dependent  clauses,  and  mainly  to  TO-infinitive  clauses. 
Usually  the  word  whose  clause  is  ellipted  is  either  a  verb  of  communication  or  cognition  such 
as  ask,  know,  force,  wish  or  an  adjective  such  as  willing,  eager,  sure. 


Eg,  Captain  Landers  did  not  want  to  reveal  the 

location  of  the  battalion,  but  the  terrorists 
forced  him  (to  reveal  the  location  of  the  battalion) . 

Eg,  When  the  captured  terrorist  was  asked  to  identify  the 
terrorist  leader,  he  was  willing  (to  identify  the 
terrorist  leader) . 

Eg,  The  peasant  did  not  want  to  set  fire  to  the  presidential 
farm  but  the  guerrillas  forced  him  (to  set  fire  to  the 
presidential  farm) . 

Eg,  When  the  guerrillas  were  asked  to  release  the  kidnapped  bank 
manager,  they  were  willing  (to  release  the  kidnapped  bank 
manager) . 


Criteria:  Demonstrated  that  it  determined  the  clause  for  the  (second)  sentence  (from  which 
the  clause  had  been  omitted),  and  correctly  interpreted  the  sentence. 

_ Score:  _ 


XI- 1 2 


XII.  SEMANTICS  OF  EVENTS 


An  event  is  an  intersection  of  three-dimensional  space  with  an  interval  of  time  in  some 
world.  This  definition  includes  both  happenings  and  states.  The  verb  in  a  sentence  generally 
expresses  the  main  concept  of  an  event.  Other  features  of  an  event,  such  as  the  participants, 
the  time,  the  place,  the  manner  in  which  the  event  happened,  any  instruments  used,  and 
how  the  participants  felt,  may  be  expressed  using  linguistic  devices  such  as  noun  phrases,  or 
adverbials. 

In  the  following  sections  we  will  categorize  and  test  these  components  of  events. 

1.  Main  Concept  of  Events 

In  English,  verbs  are  used  to  express  the  main  concept  of  an  event.  In  the  following  subsec¬ 
tions,  we  categorize  actions  and  statives  expressed  by  the  verb.  When  testing  the  following 
categories  of  verbs,  use  sentence  types  on  which  the  system  has  succeeded. 

1.1  Verbs  Of  Motion 

1.1.1  Deictic  Verbs  Of  Motion 

Deictic  motion  verbs  are  verbs  which  express  some  object’s  motion  relative  to  a  reference 
point  or  location  determined  from  the  situational  context  of  the  utterance.  This  reference 
point  is  called  the  “deictic  center For  example,  the  deictic  motion  verb  “go”  has  the  extra 
meaning  of  away  from  “here”  where  “here”  is  the  speaker’s  location.  Other  examples  of 
deictic  motion  verbs  are  “come”,  “bring”,  ‘iake”,  “approach”,  and  “leave”. 

Eg,  Did  Jones  go  to  New  York? 

Eg,  What  salespersons  came  from  Chicago? 

Eg,  Honduran  terrorists  brought  guns  into  Guatemala. 

Eg,  Terrorists  went  into  Guatemala  and  killed  the  Prime  Minister. 
Eg,  (Preliminary  Sentence:  Police  arrested  12  terrorists  at  the 
Government  House.)  Police  took  the  terrorists  to  jail. 

Eg,  Guerrillas  went  to  the  presidential  farm  and  burned  the 
buildings . 

Criteria:  Demonstrated  understanding  that  the  deictic  motion  verb  signals  movement  with 
respect  to  a  reference  point  determined  from  the  situational  context. 

_  Score:  _ 


1.1.2  Non-Deictic  Verbs  Of  Motion 

Non-deictic  motion  verbs  express  some  object’s  motion,  and  the  verb  alone  does  rot  evoke 
a  deictic  center.  For  example,  the  non-deictic  motion  verb  “move”  has  no  meaning  in  itself 
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of  a  relationship  to  a  deictic  center,  though  it  may  be  used  with  other  deictic  elements,  for 
example  “there”,  in  the  sentence: 

The  shortstop  moved  there. 

“There”  is  deictic  because  it  refers  to  a  location  known  only  in  the  situational  context  of  the 
utterance. 

Other  examples  of  non-deictic  motion  verbs  are  “run”,  “roll”,  “fall”,  “bounce ”,  “slide”,  “fly”, 
“walk”,  “ship”,  “travel”,  etc. 

Eg,  Which  departments  ship  printers? 

Eg,  Which  salesman  moved  from  New  York? 

Eg,  ARENA  flew  over  El  Salvador  and  dropped  bombs. 

Eg,  Terrorists  dropped  bombs  on  El  Salvador. 

Eg,  Honduran  terrorists  moved  to  El  Salvador  and  killed  the 
Prime  Minister. 

Eg,  Guerrillas  moved  onto  the  presidential  farm  and  burned 
the  buildings. 

Criteria:  Demonstrated  understanding  that  the  non-deictic  motion  verb  expressed  some 
object’s  movement. 

_ Score:  _ 


1.2  Verbs  Of  Mental  Phenomenon 

1.2.1  Verbs  of  Perception  (t.b.d.) 

Verbs  which  express  perception  can  be  divided  into  those  which  express  the  perception  from 
an  external,  or  public,  viewpoint  and  those  which  express  the  perception  from  an  internal, 
or  private,  viewpoint. 

1.2. 1.1  Perception  from  an  External  Viewpoint  (t.b.d.) 

By  “perception  from  an  external  viewpoint”,  we  mean  events  of  perception,  the  occurrence  of 
which  can  be  known  by  someone  outside  of  the  perceiver,  from  external  cues.  For  example, 
in 


John  sees  Mary. 

the  seeing  is  an  event  assumed  by  the  speaker  to  have  happened  from  external  cues.  Other 
verbs  in  this  category  are  “see”,  “hear”,  and  ‘Haste”. 
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1.2. 1.2  Perception  from  an  Internal  Viewpoint  (t.b.d.) 

By  “perception  from  an  internal  viewpoint”,  we  mean  events  of  perception,  the  occurrence  of 
which  cannot  be  known  by  someone  outside  of  the  perceiver,  unless  specifically  made  aware. 
For  example,  in: 

John  looks  ill. 

the  perceptual  act  is  not  visible  from  the  outside,  but  is  the  speaker’s  internal  perception. 
In: 


It  looks  like  a  spider,  to  Bill. 

the  speaker  has  presumably  been  told  of  Bill’s  perception.  Other  verbs  in  this  category  are 
“ sounds  (like)”,  and  “seems”. 


1.2.2  Verbs  of  Emotion  (t.b.d.) 

Verbs  of  Emotion  express  emotional  events  like  feel,  hurt,  or  ache. 

1.2.3  Verbs  of  Cognition  (t.b.d.) 

Cognitive  processes  include  processes  like  emotions  and  perceptions ,  treated  above,  as  well 
as  thinking,  judging,  and  learning.  Verbs  of  Cognition  express  these  latter  processes  which 
are  often  thought  of  as  being  part  of  “conscious”  experience. 


1.3  Verbs  of  Communication 

Verbs  of  expression  express  communicative  events. 

1.3.1  Illocutionary  Verbs 

Illocutionary  verbs  express  communicative  events  which  result  in  the  movement  of  informa¬ 
tion  from  speaker  to  hearer.  Examples  are  “ask”,  “say”,  ‘Hell”,  and  so  on. 

Eg.  List  the  employees. 

Eg,  Tell  me  who  sells  in  the  New  York  Region. 

Eg,  Reporters  said  that  guerrillas  killed  the  Prime  Minister. 

Eg,  Reporters  announced  guerrillas  killed  the  Prime  Minister. 

Eg,  The  government  said  that  the  guerrilla  attack  killed  one  civilian. 
Eg,  URNG  guerrillas  announced  that  they  had  attacked  the  presidential 
farm . 

Criteria:  Demonstrated  understanding  of  the  communicative  event  expressed  by  the  illocu¬ 
tionary  verb. 
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Score: 


1.3.2  Performative  Verbs 

Performatives  verbs  express  communicative  events  during  which  something  besides  simply 
the  movement  of  information  from  speaker  to  hearer  occurs.  For  example,  the  verb  “to 
convince”  expresses  a  communication  event  with  the  extra  meaning  of  someone’s  belief  being 
changed.  Other  examples  are  command,  concede,  and  so  on. 

Eg,  Who  orders  new  parts? 

Eg,  Who  claims  to  be  a  college  graduate? 

Eg,  Did  Jones  hire  John  Smith? 

Eg,  ARENA  claimed  to  have  attacked  a  town  in  El  Salvador. 

Eg,  Police  reported  that  guerrillas  killed  the  Prime  Minister. 

Eg,  The  government  accused  the  URNG  guerrillas  of  the  attack  on 
the  presidential  farm. 

Eg,  URNG  guerrillas  claimed  responsibility  for  the  attack  on  the 
presidential  farm. 

Criteria:  Demonstrated  understanding  of  both  the  communication  and  the  additional  occur¬ 
rence  expressed  by  the  performative  verb. 

_ Score:  _ 


1.4  Verbs  of  Happenings 

Verbs  of  happenings  express  events  not  covered  in  the  above  groups.  Examples  from  this 
group  are  “find”,  “kill”,  “get”,  “hit”,  “pay”,  “buy”,  “shoot”,  and  so  on. 

Eg,  Who  sells  in  New  York? 

Eg,  What  salary  does  John  Smith  earn? 

Eg,  Terrorists  killed  the  Prime  Minister. 

Eg,  Guerrillas  bombed  the  Central  Bank  of  Peru  yesterday. 

Eg,  Guerrillas  burned  the  presidential  farm. 

Eg,  The  rebels  bombed  the  Central  Bank  yesterday. 

Criteria:  Demonstrated  understanding  of  the  event  expressed  by  the  verb  of  happening. 

_ Score: _ 


EVENT  PARTICIPANTS 
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Event  participants  are  persons  or  things  which  participate  in  the  event.  There  are  different 
types  of  roles  that  entities  can  fill  with  respect  to  the  event.  These  roles  are  called  u semantic 
roles”  since  they  are  used  to  capture  the  semantics  or  meanings  of  natural  language  utterances 
or  texts.  There  may  be  more  than  one  participant  who  fills  a  certain  semantic  role.  If  the 
second  participant  fills  a  lesser  role  than  the  first  participant,  s/he  is  a  co-participant.  For 
example,  a  secondary  agent  is  a  co-agent ,  and  a  secondary  theme  is  a  co-theme.  If  two 
participants  are  recognized  as  having  an  equal  role,  then  the  “ co -”  prefix  is  not  used.  The 
latter  can  be  signaled  by  using  a  conjunctive  noun  phrase. 

Note  that  these  participants  naturally  occur  in  certain  combinations.  Therefore,  the  same 
sentence  may  be  used  to  test  a  number  of  the  following  semantic  roles. 

2.  Agency 

2.1  Agent 

An  agent  is  the  animate  being  or  group  of  such  beings  who  instigates  or  is  responsible  for 
the  action  of  an  event.  The  agent  is  generally  found  in  subject  position,  or  is  signaled  with 
a  fty-phrase  in  a  passive  sentence.  For  example, 

John  walked  the  dog. 

John  was  arrested  by  the  police. 

John  and  Joe  played  a  game. 

Eg.  List  the  salesperson  who  earns  the  highest  salary. 

Eg,  Who  manages  department  17? 

Eg,  Who  sold  the  most  printers  in  1990? 

Eg,  Terrorists  killed  the  Prime  Minister. 

Eg,  Guerrillas  bombed  the  Central  Bank  of  Peru  yesterday. 

Eg,  Guerrillas  attacked  the  presidential  farm. 

Eg,  President  Cerezo  was  kidnapped  by  URNG  guerrillas. 

Criteria:  Correctly  determined  the  agent  and  its  responsibility  for  the  action  or  its  role  in 
instigating  the  action  expressed  by  the  sentence. 

_ Score:  _ 


2.2  Co-Agent 

A  co-agent  is  the  animate  being  who  performs  the  action  along  with  the  agent,  but  in  a 
lesser,  subsidiary  role.  This  case  is  signaled  by  the  use  of  a  with- phrase.  A  co-agent  only 
occurs  in  a  sentence  with  an  agent. 

John  walked  the  dog  with  Jim. 
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Eg,  Who  manages  department  17  with  John  Smith? 

Eg,  Who  sells  with  John  Doe? 

Eg,  ARENA  attacked  a  village  vith  a  group  of  armed  men. 

Eg,  Shining  Path  bombed  an  electric  facility  with  unidentified 
individuals . 

Eg,  The  URNG  guerrilla  leader  attacked  the  presidential  farm  with  an 
unidentified  companion. 


Criteria:  Correctly  determined  the  co-agent  and  its  responsibility  for  the  action  or  its  role 
in  instigating  the  action  expressed  by  the  sentence. 

_ Score: _ 


3.  Experiencer 

An  experiencer  is  the  animate  being  who  is  passively  involved  in  event  of  perception  or  in 
a  psychological  event.  Certain  psychological  events,  for  example  thinking,  require  an  active 
animate  being.  Others,  such  as  notice,  are  more  accidental.  For  example, 

John  noticed  a  rose-breasted  grossbeak. 


Eg,  Who  heard  that  Joe  Smith  is  getting  a  raise? 

Eg,  Employees  saw  guerrillas  bomb  the  Telephone  Company 
Building. 

Eg,  Peasants  heard  terrorists  attack  a  town  in  Peru  yesterday. 

Eg,  Guards  saw  guerrillas  attack  the  presidential  farm. 

Eg,  The  peasants  heard  guerrillas  blow  up  the  tank  truck. 

Criteria:  Correctly  determined  the  experiencer  and  its  involvement  in  the  event  of  perception 
or  in  the  psychological  event  expressed  by  the  sentence. 

_ Score: _ 


4.  Beneficiary 

The  beneficiary  is  the  animate  being  for  whom  a  certain  event  is  performed. 

I  painted  Susan  a  picture. 

I  painted  a  picture  for  Susan. 

Eg,  Who  worked  for  X  corporation  in  1989? 

Eg,  Who  sells  for  Xidec? 
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Eg,  Who  pays  John  Smith? 

Eg,  Employees  bombed  the  Telephone  Company  Building  for  ARENA. 

Eg,  Peasants  kidnapped  the  Prime  Minister  for  Shining  Path. 

Eg,  The  URNG  guerrillas  burned  the  presidential  farm  for  their  leader. 
Eg.  THE  URNG  guerrillas  stole  food  for  their  families. 

Criteria:  Correctly  determined  the  beneficiary  and  its  role  as  the  animate  being  for  whom 
the  event  expressed  by  the  sentence  is  performed. 

_ Score: _ 


5.  Patient 

A  patient  is  the  animate  being  who  is  acted  upon,  or  who  is  undergoing  some  change.  For 
example, 

The  Mayor  was  assassinated. 

John  kissed  Mary. 

Eg,  Which  department  hired  John  Smith? 

Eg,  When  was  Smith  promoted? 

Eg,  The  Prime  Minister  was  assassinated. 

Eg,  A  terrorist  group  assassinated  the  Prime  Minister. 

Eg,  The  guerrilla  attack  on  the  presidential  farm  killed  one  security 
guard . 

Eg,  The  URNG  guerrillas  kidnapped  President  Cerezo. 

Criteria:  Correctly  determined  the  patient  and  demonstrated  understanding  that  some  was 
action  is  performed  upon  it  or  that  it  undergoes  some  change. 

_ _ _ Score: _ 


6.  Theme 

A  theme  is  a  non-animate  object  being  acted  upon,  or  undergoing  some  change.  For  example, 
the  window  is  the  theme  in  the  next  sentence,  where  it  appears  in  object  position: 

John  broke  the  window. 


The  theme  of  a  sentence  may  also  appear  in  subject  position.  In  the  next  sentence  the 
window  is  also  the  theme: 
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The  window  broke. 


6.1  Theme  in  Object  Position 

Test  for  the  theme  in  object  position  (the  position  following  the  verb). 

Eg,  Who  manages  department  17? 

Eg,  List  the  salesperson  who  earns  the  highest  salary? 

Eg,  Guerrillas  bombed  the  Central  Bank  of  Peru  yesterday. 

Eg,  Guerrillas  burned  the  presidential  farm. 

Eg,  The  rebels  bombed  the  Central  Bank. 

Criteria:  Correctly  determined  the  theme  in  object  position  and  demonstrated  understanding 
that  some  action  is  performed  upon  it  or  that  it  undergoes  some  change. 

_ Score: _ 


6.2  Theme  in  Subject  Position 

Test  for  the  theme  in  subject  position  (the  initial  position  in  the  sentence). 

Eg,  CPUs  are  sold  by  which  office? 

Eg,  The  Telephone  Company  Building  exploded  yesterday. 

Eg,  The  electric  facility  was  bombed  by  ARENA. 

Eg,  The  presidential  farm  was  burned  by  guerrillas. 

Eg,  The  presidential  farm  burned  in  yesterday’s  attack. 

Criteria:  Correctly  determined  the  theme  in  subject  position  and  demonstrated  understand¬ 
ing  that  some  action  is  performed  upon  it  or  that  it  undergoes  some  change. 

_ Score:  _ 


6.3  Co-Theme 

In  sentences  where  there  are  two  or  more  actions  each  with  a  theme,  or  in  events  where  there 
is  an  exchange  of  objects,  tangible  or  otherwise,  the  secondary  theme  (in  the  subordinate 
clause  or  phrase)  is  the  co-theme.  A  co-theme  only  occurs  in  a  sentence  which  has  a  theme. 
The  secondary  theme  is  the  physics  book  in  the  following  sentence: 

Joe  gave  John  $20  for  a  physics  book. 

Eg,  Who  sells  vacuum  cleaners  for  $300? 

Eg,  Who  works  a  40  hour  week  for  $500? 
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Eg.  Terrorists  Kidnapped  the  Prime  Minister  for  a  $74,000 
ransom. 

Eg,  ARENA  traded  guns  for  bombs  with  Shining  Path. 

Eg,  Guerrillas  attacked  the  presidential  farm  and  stole  the 
the  trucks. 


Criteria:  Correctly  determined  the  co-theme  and  demonstrated  understanding  that  some 
action  is  performed  upon  it  or  that  it  undergoes  some  change. 

_ _ _ _ _ _  Score: _ 


7.  Source 

Source  is  the  animate  or  inanimate  entity  from  which  an  object  is  transferred.  The  object  can 
be  either  tangible  or  intangible.  An  animate  source  is  sometimes  called  the  “from-possessor” 
and  is  the  original  possessor  of  the  object.  For  example,  Joe  is  the  source  in  the  following: 

John  got  a  physics  book  from  Joe. 

An  inanimate  source  is  generally  the  entity  from  which  something  is  transferred.  The  transfer 
may  be  a  transfer  of  information,  ideas,  possession,  ownership,  and  so  on,  and  may  involve 
no  movement  in  any  spatial  dimension. 

There  is  often  an  overlap  of  thematic  roles.  For  example,  in  the  following  sentence  Joe  is 
both  the  agent  and  the  source: 

Joe  gave  John  $20. 

Eg,  Which  department  pays  John  Smith? 

Eg,  From  which  office  does  John  Smith  sell  printers? 

Eg,  Terrorists  stole  $25,000  from  the  Central  Bank  of  Peru. 

Eg,  Peasants  attacked  the  Peruvian  Embassy  with  guns  from 
the  Soviet  Union. 

Eg,  Guerrillas  stole  food  from  peasants  in  Flores. 


Criteria:  Correctly  determined  the  source  and  its  role  of  being  the  animate  or  inanimate 
entity  from  which  an  object  is  transferred. 

_  Score:  _ 


8.  Goal 
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Goal  is  the  animate  or  inanimate  entity  to  which  an  object  is  transferred.  The  object  can 
be  either  tangible  or  intangible.  An  animate  goal  is  sometimes  called  ti  'Ho- possessor”  and 
is  the  final  possessor  of  the  object.  For  example,  Joe  is  *he  goal  in  b».th  of  the  following 
examples: 

Joe  got  a  physics  book  from  Joan. 

John  gave  a  physics  book  to  Joe. 

An  inanimate  goal  is  generally  the  entity  to  which  something  is  transferred,  for  example,  in: 

John  donated  $20  to  Muscular  Dystrophy  Research. 

The  transfer  may  be  a  transfer  of  information,  ideas,  possession,  ownership,  and  so  on,  and 
may  involve  no  movement  in  any  spatial  dimension.  The  destination  of  the  movement  of 
some  object  along  a  spatial  path  is  not  necessarily  a  goal. 

Eg,  Which  salesperson  transferred  to  the  Chicago  office? 

Eg,  Terrorists  robbed  the  Central  Bank  of  Peru  and  gave 
the  money  to  ARENA. 

Eg,  Guerrillas  smuggled  guns  to  peasants  in  El  Salvador. 

Eg.  Terrorists  fired  on  peasants  in  Peru. 

Eg,  Guerrillas  gave  guns  to  Honduran  rebels  who  attacked  the 
presidential  farm. 

Criteria:  Correctly  determined  the  goal  and  its  role  of  being  the  animate  or  inanimate  entity 
to  which  an  object  is  transferred. 

_ _ _ Score: _ 


9.  Instrument 

An  instrument  is  a  tool  or  material  used  to  perform  some  action.  Instrument  can  be  ex¬ 
pressed: 

•  Using  a  noun  phrase: 

The  hammer  broke  the  window. 

John  broke  the  window  with  the  hammer. 

•  Using  a  verb  which  explicitly  names  the  instrument: 

Jack  hammered  the  nail. 

The  villagers  stoned  the  woman. 
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•  Using  a  verb  in  which  the  instrument  is  not  explicit.  For  example,  the  verb  kick  always 
implies  a  foot  as  a  covert  instrument. 

Ising  either  a  noun  phrase  or  a  explicit  verb,  test  for  use  of  an  instrument  in  a  sentence. 

Eg,  Which  salesperson  phones  orders  to  the  Chicago  office? 

Eg.  Terrorists  attacked  the  Central  bank  in  Peru  with  bombs. 

Eg,  Terrorists  bombed  the  Central  bank  in  Peru. 

Eg,  Guerrillas  attacked  the  presidential  farm  with  guns. 

Criteria:  Correctly  determined  the  instrument  and  its  role  as  the  tool  or  material  used  to 
erform  the  action  expressed  by  the  sentence. 

_ Score: _ 


0.  Participant  Combinations 

i  the  following  sections,  we  will  test  the  common  participant  combinations. 

0.1  Agent  and  Theme 

>ur  examples  show  the  agent  in  subject  position  and  the  theme  in  object  position. 

Eg,  List  the  people  who  earn  $15,000. 

Eg,  Who  sells  CPUs? 

Eg,  Employees  bombed  the  Telephone  Company  Building. 

Eg,  Terrorists  attacked  the  electric  facility. 

Eg.  Guerrillas  attacked  the  presidential  farm. 

riteria:  Correctly  determined  the  agent  and  the  theme  and  demonstrated  understanding  of 
leir  roles  as  instigator  and  undergoer,  respectively,  of  the  event  expressed  by  the  sentence. 

_ Score: _ 


D.2  Agent,  Theme  and  Instrument 

ur  examples  show  the  agent  in  subject  position,  the  theme  in  object  position,  and  the 
istrument  signaled  by  a  “with”,  or  a  “using”. 

Eg,  Who  sells  CPUs  using  a  telephone? 

Eg,  Terrorists  attacked  the  Telephone  Company  Building  with 
a  lightplane. 

Eg,  Maoists  attacked  the  electric  facility  with  machineguns. 

Eg,  Guerrillas  attacked  the  presidential  farm  with  guns. 
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Criteria:  Correctly  determined  the  agent ,  the  theme ,  and  the  instrument  and  demonstrated 
understanding  of  their  roles  as  instiga*  undergoer,  and  tool  or  material,  respectively,  of 
the  event  expressed  by  the  sentence. 

_ Score: _ 


10.3  Agent,  Patient  and  Instrument 

Our  examples  show  the  agent  in  subject  position,  the  patient  in  object  position,  and  the 
instrument  signaled  by  a  “with”,  or  a  “using”. 

Eg,  John  Smith  sold  CPUs  to  customers  using  a  telephone. 

Eg,  Terrorists  killed  the  Prime  Minister  with  a  gun. 

Eg,  ARENA  shot  the  Mayor  with  a  submachinegun . 

Eg,  Guerrillas  attacked  the  presidential  farm  with  a  machinegun. 

Criteria:  Correctly  determined  the  agent,  tht  patient,  and  the  inst-ument  and  demonstrated 
understanding  of  their  roles  as  instigator,  undergoer,  and  tool  or  material,  respectively,  of 
the  event  expressed  by  the  sentence. 

_ Score:  _ 


10.4  Source,  Goal,  Theme  and  Co-theme 

In  events  where  there  is  an  exchange  of  objects,  we  can  think  of  the  event  as  having  two 
sub-events,  each  being  a  transfer  of  an  object  from  a  source  to  a  goal.  Test  a  sentence  which 
describes  this  type  of  event. 

Eg,  Did  the  New  York  office  trade  John  Doe  to  the  Boston  office 
for  Fred  Smith? 

Eg,  Who  sold  printers  to  customers  for  $400? 

Eg,  Guerrillas  traded  guns  for  food  with  peasants  in  El  Salvador. 

Criteria:  Correctly  determined  the  source,  goal,  theme,  and  co-theme  and  demonstrated 
understanding  of  their  roles  in  the  exchange  event  expressed  by  the  sentence. 

_ Score: _ 


OTHER  SEMANTIC  ROLES 

There  are  a  number  of  other  semantic  roles  aside  from  participants.  These  additional  roles 
are  frequently  expressed  using  adverbials. 
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[.  Spatial  Information  of  Events 

►atial  information  can  be  categorized  as  follows: 

..1  Position 

ysition  is  where  an  event  occurs: 

He  lay  on  the  bed. 

They  were  walking  in  the  park. 


Eg,  Who  works  in  New  York? 

Eg.  What  employees  work  in  department  17? 

Eg,  Guerrillas  assassinated  the  Prime  Minister  in  El  Salvador. 
Eg,  Terrorists  attacked  civilians  at  the  Government  House. 

Eg,  Guerrillas  killed  a  security  guard  at  the  presidential  farm. 


‘iteria:  Demonstrated  understanding  of  the  spatial  position  involved  in  the  event  expressed 
’  the  sentence. 

_ Score: _ 


.2  Path 

ith  includes  several  subparts,  including 

•  direction: 

They  drove  westwards. 

She  walked  around  the  park. 

•  origin: 

She  was  walking  from  the  store. 

The  race  started  at  the  store. 

•  destination: 

She  was  walking  to  the  store. 

The  race  ended  at  the  highway. 

ith  is  often  incorporated  into  verbs  of  motion,  for  example  in:  “enter”  (go  into),  “exit”  (go 
t  from),  “ascend”  (go  up),  “descend”  (go  down),  and  “circumnavigate”  (go  around). 
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Eg,  Was  Jones  transferred  to  Chicago? 

Eg,  Did  Smith  move  from  the  New  York  Office? 

Eg,  Terrorists  attacked  targets  in  a  path  from  Peru  to  Colombia. 

£g.  Terrorists  went  to  Peru  and  killed  12  civilians. 

Eg,  Guerrillas  went  to  the  presidential  farm  and  killed  a  security 
guard. 

Eg.  Guerrillas  brought  guns  to  the  presidential  farm  and  attacked. 

Criteria:  Demonstrated  understanding  of  the  path  involved  in  the  event  expressed  by  the 
sentence. 

_  Score:  _ 


11.3  Distance 

Distance  is  linear  spatial  measure: 

They  traveled  a  long  way. 

She  drove  fifty  kilometers. 

Eg,  How  many  miles  did  Smith  travel  last  month? 

Eg,  Did  Jones  move  500  miles  to  Chicago? 

Eg,  Terrorists  traveled  500  kilometers  and  killed  12  civilians. 

Eg,  Terrorists  went  200  kilometers  to  kill  the  Prime  Minister. 

Eg,  Guerrillas  drove  fifty  miles  to  the  presidential  farm  and 
burned  the  buildings. 

Criteria:  Demonstrated  understanding  of  the  distance  involved  in  the  event  expressed  by  the 
sentence. 

_ _ _ Score:  _ 


12.  Temporal  Information  of  Events 

Time  is  often  conceptualized  as  a  position  on  or  a  movement  along  a  time  scale.  Temporal 
issues  are  also  covered  in  the  section  titled  “Temporal  Aspect  of  Verb  Phrases”. 

Temporal  information  can  be  categorized  as  follows: 

12.1  Position 

Position  is  the  point  or  interval  on  the  relevant  time  scale: 

She  drove  to  Chicago  on  Sunday. 

He  was  there  last  week.  [Quirk  et  al  1985,  p.  481] 
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Eg,  Who  worked  in  1989? 

Eg.  List  the  men  who  worked  in  December. 

Eg,  Yesterday  terrorists  bombed  the  Central  Bank  in  Cuba. 

Eg.  On  December  2,  guerrillas  kidnapped  the  Prime  Minister. 
Eg,  Guerrillas  attacked  the  presidential  farm  on  2  February. 


Criteria:  Demonstrated  understanding  of  the  temporal  position  on  some  time  scale  for  the 
event  expressed  by  the  sentence. 

_ Score: _ 


12.2  Duration 

Duration  may  be  talked  about  in  terms  of: 

•  time  length,  which  is  unspecified  as  to  direction: 

I  am  staying  three  weeks.  [Quirk  et  al  1985,  p.  482] 

I  wrote  for  an  hour. 

•  forward  span,  called  temporal  destination: 

I’ll  be  here  until  next  week. 

•  backward  span,  called  temporal  origin: 

I’ve  been  here  since  Saturday. 

Eg,  Who  has  worked  for  3  years? 

Eg,  List  the  employees  who  have  worked  since  1984. 

Eg,  Terrorists  bombed  the  Central  Bank  in  Cuba  for  2  hours. 

Eg,  Terrorists  have  attacked  the  Government  House  since  December  25. 
Eg,  Guerrillas  attacked  on  December  12  and  bombed  the  Government 
House  for  5  days. 

Eg,  The  government  has  reported  two  guerrilla  attacks  on  the 
presidential  farm  since  1  February. 


Criteria:  Demonstrated  understanding  of  the  duration  of  the  event  expressed  by  the  sentence. 
_ Score: _ 


12.3  Frequency 

Frequency  is  elicited  by  the  question  “ How  often  . ..?” 
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I  wash  my  car  frequently /weekly/ once  a  month/as  often  as  possible. 

Eg,  Who  is  paid  weekly? 

Eg,  Who  is  paid  biweekly? 

Eg,  What  salesperson  earned  a  commission  every  month  in  1989? 

Eg,  Terrorists  bombed  the  Central  Bank  in  Cuba  every  two  hours 
yesterday. 

Eg,  In  December,  guerrillas  kidnapped  a  Government  Official 
every  day. 

Eg,  Guerrillas  kidnapped  a  Government  Official  three  times  in 
December . 

Eg,  Guerrillas  have  attacked  the  presidential  farm  twice  since 
February  1. 

Criteria:  Demonstrated  understanding  of  the  frequency  of  the  event  expressed  by  the  sen¬ 
tence. 


Score: 


12.4  Relationship 

Relationship  expresses  the  relation  of  the  time  frame  of  one  event  to  the  time  frame  of 
another: 

He  had  visited  his  mother  already  when  I  saw  him  yesterday.  [Quirk  et  al  1985, 
p.  482] 

Eg,  List  the  employees  who  sold  while  they  worked  in  New  York. 

Eg,  Who  managed  department  17  before  John  Smith  was  hired? 

Eg,  Terrorists  bombed  the  Government  House  after  ARENA  killed 
the  Prime  Minister. 

Eg,  Guerrillas  killed  a  Government  Official  during  an  attack 
on  the  Government  House. 

Criteria:  Demonstrated  understanding  of  the  relationship  of  the  time  frame  of  two  events  as 
expressed  by  the  sentence. 

- Score: _ 


13.  Participant  Combination  with  Time  and  Location 

Combine  agent,  theme  and  instrument  with  time  and  location.  Our  examples  show  the 
agent  in  subject  position,  and  the  theme  in  object  position  and  propositional  phrases  for  the 
instrument,  the  time,  and  the  location. 
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Eg,  Who  sold  CPUs  using  a  telephone  in  January  at  the  New  York 
office? 

Eg,  Who  sold  CPUs  in  January  at  the  New  York  office  using  a 
telephone? 

Eg,  Terrorists  attacked  the  Telephone  Company  Building  in 
Nicaragua  with  a  lightplane  on  December  12. 

Eg,  In  El  Salvador  on  November  26,  ARENA  attacked  the  Government 
House  with  machineguns. 

Eg,  Guerrillas  attacked  the  president  with  machineguns  on  February 
2  in  Flores. 

Criteria:  Correctly  determined  the  agent ,  theme ,  and  instrument  and  demonstrated  under¬ 
standing  of  their  roles  as  instigator,  undergoer,  and  tool  or  material,  respectively,  of  the 
event  expressed  by  the  sentence.  Also  demonstrated  understanding  of  the  temporal  position 
and  spatial  position. 

_ _  Score: _ 


14.  Manner 

Manner  is  the  manner  or  style  in  which  an  action  is  carried  out. 

He  prayed  fervently.  [Quirk  et  al  1985,  p.  558] 

She  spoke  with  cold  deliberation. 

Manner  is  often  expressed  by  a  motion  verb  which  conflates  an  event  of  motion  with  the 
manner  or  style  of  that  motion  [Talmy  1975]: 

The  ball  rolled/bounced/slid  down  the  hall. 

Eg,  What  salesperson  works  productively? 

Eg,  Does  Smith  manage  department  12  efficiently? 

Eg,  Terrorists  efficiently  bombed  the  Central  Bank  in  Cuba. 

Eg,  Guerrillas  ruthlessly  murdered  the  Prime  Minister. 

Criteria:  Demonstrated  understanding  of  the  manner  in  which  the  action  expressed  by  the 
sentence  is  carried  out. 

_ _ _ _ _ Score: _ 


15.  Means 

Means  is  the  technique  or  method  by  which  an  action  is  carried  out: 
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He  decided  to  treat  the  patient  surgically. 

I  go  to  school  by  car.  [Quirk  et  al  1985,  p.  559] 

(Means  is  often  implicit  in  the  verb  chosen  to  express  an  event.  For  instance,  in: 

The  doctor  operated  on  the  patient. 

the  means  is  also  surgically,  as  above.) 

Eg,  Who  was  rewarded  for  high  sales  with  a  pay  raise? 

Eg,  Who  is  paid  by  commission? 

Eg,  Guerrillas  destroyed  the  Peruvian  Embassy  with  an  explosion. 
Eg,  Guerrillas  destroyed  the  Peruvian  Embassy  by  bombing  it. 

Eg,  The  rebels  destroyed  the  bank  by  burning  it. 

Criteria:  Demonstrated  understanding  of  the  means  by  which  the  action  expressed  by  the 
sentence  is  carried  out. 

_ _ _ _ Score: _ 


16.  Respect 

Respect  is  an  abstract  semantic  category  which  is  concerned  with  identifying  a  point  of 
reference,  or  attitude,  with  respect  to  which  the  clause  being  qualified  derives  its  truth 
value. 

For  example,  the  next  sentence  refers  first  to  location,  then  to  time,  as  the  abstract  respect: 

So  far  as  traveling  facilities  are  concerned,  we  have  obviously  made  a  popular 
decision;  but  with  respect  to  *he  date,  many  people  are  expressing  dissatisfaction. 
[Quirk  et  al  1985,  p.  438] 

Eg,  With  respect  to  education,  who  is  the  highest  qualified 
employee? 

Eg,  Regarding  1990,  which  salesperson  sold  the  most  vacuum 
cleaners? 

Eg,  Regarding  the  terrorist  attack  on  Peru,  seven  civilians 
were  killed. 

Eg,  Concerning  civilians,  the  ARENA  bombing  in  December  resulted 
in  12  deaths. 

Eg,  Regarding  the  guerrilla  attack  on  the  presidential  farm,  one 
security  guard  was  killed. 
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Criteria:  Demonstrated  understanding  of  the  respect  that  is  used  to  derive  the  truth  value 
of  the  main  sentence. 


Score: 


17.  Contingency 

Contingency  refers  to  the  possible  dependency  relationships  between  two  events,  for  example 
cause  or  result.  One  of  the  events,  usually  that  expressed  by  the  main  clause  of  the  sentence, 
is  felt  to  be  a  more  central  part  of  a  sentence.  The  contingency  relationship  describes  the 
lesser  event’s  relationship  to  this  central  event  and  may  be  categorized  as  in  the  following 
subsections: 

17.1  Cause 

Cause  is  the  type  of  relationship  in  which  an  event  causes  or  motivates  what  we  conceptualize 
as  the  central  event  (expressed  by  the  verb)  in  the  sentence: 

She  died  of  cancer. 

Cause  may  also  be  expressed  by  an  inanimate  agent  in  the  sentence’s  subject  position: 
Cancer  killed  her. 

Eg,  Jane  Smith  earned  the  highest  commission  because  of  high  sales. 
Eg,  The  Prime  Minister  died  from  gunshot  wounds. 

Eg,  Because  of  yesterday's  bombing  in  Peru,  12  people  died. 

Eg,  The  guerrilla  attack  on  the  presidential  farm  killed  a  security 
guard . 

Criteria:  Demonstrated  understanding  of  the  causal  relationship  between  a  sub-event  (the 
cause)  and  the  primary  event  expressed  by  the  sentence. 

_ Score: _ 


17.2  Reason 

A  reason  is  the  rationale  for  the  central  event: 

He  bought  the  book  because  of  his  interest  in  metaphysics.  [Quirk  et  al  1985,  p. 
484] 


Eg,  Jane  Smith  is  a  salesperson  to  send  her  children  to  college. 
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Eg,  Who  moved  to  Nev  York  because  she  got  married? 

Eg.  List  the  people  who  were  promoted  because  they  had  high  sales. 
Eg,  Terrorists  attacked  the  Peruvian  Embassy  because  of  food 
shortages . 

Eg,  Because  of  food  shortages,  terrorists  attacked  the  Peruvian 
Embassy. 

Eg,  Guerrillas  killed  the  security  guard  because  he  shot  a 
guerrilla  during  the  attack. 


Criteria:  Demonstrated  understanding  of  the  rationale  relationship  between  a  sub-event  (the 
reason)  and  the  primary  event  expressed  by  the  sentence. 

_ Score: _ 


17.3  Purpose 

Purpose  is  the  desired  result  of  the  sentence’s  central  event.  For  example: 
He  bought  the  book  so  he  could  study  metaphysics. 


Eg,  Who  moved  to  Nev  York  because  of  a  possible  promotion? 

Eg,  List  the  employees  who  attended  college  so  they  could  become 
managers . 

Eg,  Terrorists  kidnapped  the  Prime  Minister  so  that  they  could 
kill  him. 

Eg,  To  gain  the  release  of  five  prisoners.  Shining  Path  killed 
twelve  peasants. 

Eg,  Guerrillas  attacked  the  presidential  farm  to  kidnap 
President  Cerezo. 


Criteria:  Demonstrated  understanding  that  the  sub- event  (the  purpose )  is  the  desired  result 
of  the  primary  event  expressed  by  the  sentence. 

_ Score: _ 


17.4  Result 

Result  is  the  relationship  such  that  there  is  a  subordinate  event  which  directly  results  from 
the  central  event  in  the  sentence.  For  example: 

He  read  the  book  carefully,  and  so  he  acquired  some  knowledge  of  metaphysics. 
[Quirk  et  al,  1985,  p.  484] 
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Eg,  Who  moved  to  Nev  York  and  so  received  a  promotion? 

Eg.  List  the  employees  who  earned  an  MBA  and  therefore  were  made 
managers . 

Eg,  Terrorists  attacked  the  Government  House  and  as  a  result  the 
Prime  Minister  was  killed. 

Eg,  The  Prime  Minister  was  killed  as  a  result  of  the  terrorist 
attack  on  the  Government  House. 

Eg,  Shining  Path  bombed  a  bank  and  thereby  killed  30  people. 

Eg,  As  a  result  of  the  guerrilla  attack  on  the  presidential  farm, 
one  security  guard  died. 


Criteria:  Demonstrated  understanding  that  the  sub-event  (the  result)  directly  results  from 
the  primary  event  expressed  by  the  sentence. 

_ Score: _ 


17.5  Condition 

Condition  is  the  relationship  such  that  the  subordinate  event  occurs  in  time  prior  to  the 
central  event  and  sets  the  condition  for  the  central  event,  as  in: 

If  he  reads  the  book  carefully,  he  will  acquire  some  knowledge  of  metaphysics. 
[Quirk  et  al,  1985,  p.  484] 

Who  will  receive  a  promotion  if  they  earn  an  MBA  ? 


Eg,  If  a  person  has  been  employed  for  20  years,  then  the  person  is 
a  senior  employee. 

Eg,  Who  will  earn  $30,000  if  they  receive  a  10%  raise? 

Eg,  If  terrorists  bomb  the  Government  House,  the  army  will  attack 
them. 

Eg,  If  people  die,  police  will  say  that  the  guerrillas  killed  them. 

Eg,  If  the  army  kills  people,  police  say  that  the  guerrillas  killed 
the  people. 

Eg.  If  the  president  dies,  the  government  will  charge  the  URNG 
guerrillas  with  murder. 


Criteria:  Demonstrated  understanding  that  the  sub-event  (the  condition)  occurs  prior  to  and 
sets  the  condition  for  the  primary  event  expressed  by  the  sentence. 

_ Score: _ 


17.6  Concession 
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Concession  is  the  relationship  such  that  the  subordinate  event  would  be  expected  to  enable 
the  central  event,  except  the  expectation  failed.  For  example,  in: 

In  spite  of  his  enthusiasm, ,  he  didn’t  win.  [Quirk  et  al,  1985,  p.  485] 


the  expectation  is  that  a  large  amount  of  enthusiasm  would  lead  to  the  event  of  winning, 
and  that  expectation  failed.  In  the  sentence: 

Who  will  receive  a  demotion  even  if  they  earn  an  MBA  ? 

the  expectation  is  that  earning  a  MBA  does  not  lead  to  demotions,  but  in  this  case  the 
expectation  fails. 

Eg,  List  the  salespersons  who  will  not  earn  a  commission  in  spite 
of  a  10%  sales  increase. 

Eg,  Even  if  terrorists  do  not  bomb  the  Government  House,  the  terror¬ 
ists  will  bomb  the  bank. 

Eg.  Even  if  terrorists  bomb  the  Government  House,  the  terrorists  will 
bomb  the  bank. 

Eg,  Even  if  people  do  not  die,  police  will  say  that  the  guerrillas 
killed  people. 

Eg,  Even  if  the  guerrillas  kill  people,  they  say  that  the  police 
killed  them. 

Eg,  Even  if  the  president  had  not  died,  the  URNG  guerrillas 
would  have  been  arrested  for  the  attack. 

Criteria:  Demonstrated  understanding  that  the  sub-event  (the  concession)  is  expected  to 
set  the  condition  for  the  primary  event  expressed  by  the  sentence,  however  the  expectation 
failed. 


Score: 


18.  Degree 


Degree  is  the  grade  or  position  on  the  scale  upon  which  the  action  expressed  by  the  verb  is 
measured.  For  example, 


the  action  expressed  by: 
like 
hurt 
help 
want 


is  measured  on  the  scale: 
subjective  pleasure 
injury 
helpfulness 
desire 


Degree  may  be  broken  into  three  categories,  discussed  in  the  following  subsections. 
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18.1  Amplification  (t.b.d.) 

Amplification  heightens  the  degree  on  the  gradability  scale  upon  which  the  action  expressed 
by  the  verb  is  measured.  For  example, 

I  badly  want  a  drink.  [Quirk  et  al,  1985,  p.  486] 

18.2  Diminution  (t.b.d) 

Diminution  lowers  the  degree  on  the  gradability  scale  upon  which  the  action  expressed  by 
the  verb  is  measured.  For  example, 

She  helped  him  a  little  with  his  book.  [Quirk  et  al,  1985,  p.  486] 

18.3  Measure  (t.b.d) 

Measure  is  the  measure  on  the  gradability  scale  upon  which  the  action  expressed  by  the  verb 
is  measured,  when  there  is  no  implication  that  the  degree  is  high  or  low.  For  example, 

He  likes  playing  squash  more  than  his  sister  does.  [Quirk  et  al,  1985,  p.  486] 


19.  Modality 

Modality  expresses  possibility,  or  less  than  complete  knowledge  in  the  truth  of  an  event.  In 
cases  where  the  truth  of  an  event  is  not  certain,  the  speaker  expresses  his/her  degree  of 
uncertainty  by  using  modality.  Modality  is  often  expressed  using  adverbs,  as  in: 

She  probably  enjoys  living  in  Florida. 

Modality  is  also  expressed  using  modal  auxiliary  verbs,  for  example  “will”,  “may”,  “must”, 
“might”  and  so  on,  as  in: 

She  may  enjoy  living  in  Florida. 


Eg,  List  the  employees  who  possibly  will  receive  a  raise 
in  1991. 

Eg,  Who  will  definitely  receive  a  bonus  this  year? 

Eg,  Terrorists  probably  killed  the  Prime  Minister. 

Eg,  It  is  possible  that  ARENA  bombed  the  Government  House. 
Eg,  URNG  guerrillas  probably  attacked  the  La  Eminencia  farm. 
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Criteria:  Demonstrated  understanding  of  the  modality  of  the  sentence. 

_ Score: _ 

20.  Evidential 

Evidentials  axe  the  basis  on  which  the  hearer  is  to  assess  the  truth  of  a  sentence. 

It  is  generally  assumed  that  a  speaker  speaks  true  utterances.  In  cases  where  the  truth  of 
a  proposition  is  less  than  completely  reliable,  the  doubtful  proposition  is  expressed  by  an 
embedded  sentence,  in  the  main  clause,  as  in  the  following: 

Reporters  said  today  that  <proposition>. 

I  heard  (that)  <proposition> . 

Eg,  According  to  the  sales  manager,  who  is  the  best  salesman? 

Eg,  ARENA  claimed  to  have  attacked  a  town  in  El  Salvador. 

Eg,  Police  reported  that  guerrillas  killed  the  Prime  Minister. 

Eg,  URNG  guerrillas  claim  to  have  attacked  the  Santo  Tomas  farm. 

Criteria:  Demonstrated  understanding  of  the  evidential,  the  basis  on  which  the  hearer  is  to 
assess  the  truth  of  a  sentence. 

_ Score: _ 
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APPENDIX  B 

Example  NLP  System  Profile 


his  appendix  contains  an  example  profile  for  an  NLP  system  produced  by  the  NLP  Eval- 
ition  Tool  profiling  facility.  This  profile  was  produced  as  part  of  the  final  assessment  task 
hen  each  of  four  evaluators  applied  the  Tool  to  two  NLP  systems.  Although  this  profile 
iows  all  levels  of  the  classification  hierarchy,  profiles  can  also  be  generated  that  show  only 
ie  top  N  levels,  where  N  =  1,  2,  3,  etc. 
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Evaluation  Profile:  All  Levels 
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APPENDIX  C 

Summary  of  Assessment  Questionnaire  Responses 


This  appendix  provides  a  summary  of  the  responses  from  the  Interface  Technologists  to  the 
Assessment  Questionnaire  concerning  the  NLP  Evaluation  Tool.  The  Questionnaire  was 
completed  by  each  Technologist  after  he/she  finished  applying  the  Tool  to  all  of  the  NLP 
systems. 
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APPENDIX  B.  Summary  of  Assessment  Questionnaire  Responses 

1.  Ease  of  Use 

IT  #1:  Long  chapters  are  very  “tedious.”  Should  consider  breaking  it  up  into  smaller 
chapters. 

IT  #2:  Clearer  than  last  time. 

IT  #3:  Easy  to  use,  but  “tedious.”  Less  exhaustive  testing  would  make  the  tool  easier  to 
use. 

IT  #4:  The  Tool  is  clear  and  well  laid  oui  Instructions  are  unambiguous  and  test  inputs 
are  easy  to  generate. 

2.  Helpfulness  of  Examples 

IT  #1:  Examples  are  useful,  but  “too  specific  for  the  good  of  the  tool.”  They  should  be 
general  enough  to  apply  to  all  domains.  Also,  more  general  examples  would  facilitate 
a  wider  range  of  inputs  to  the  NLP  system. 

IT  #2:  They  are  helpful  for  “semi-automated  evaluation”  in  which  examples  are  used  as 
inputs.  When  used  this  way,  the  diacritical  markers  like  []  are  cumbersome. 

IT  #3:  They  are  okay.  I  used  them  in  forming  inputs. 

IT  #4:  “Examples  were  invaluable.”  They  keep  interpretation  of  test  items  consistent 
across  evaluators.  They  provide  an  “easily  graspable  form  of  what  the  Tool  was  strug¬ 
gling  to  say.”  They  also  save  time  because  they  avoid  time  that  would  be  needed  to 
pour  over  the  glossary. 

3.  Face  Validity  of  Scores 

IT  #1:  Not  always.  Sometimes  the  system  gets  “lucky,”  especially  when  the  system  is  able 
to  use  keywords.  Having  three  tries  helped  face  validity  better  for  the  database  NLP 
system. 

IT  #2:  Kind  of,  but  there  is  still  a  lot  of  judgement  required. 

IT  #3:  Seemed  to  be  adequate. 

IT  #4:  In  terms  of  semantics,  “yes”  for  database  system  ,  but  “no”  for  the  system  that 
used  keywords. 
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4.  Scoring  Criteria 


IT  #1:  The  criteria  and  scores  (S,C,P,F,  and  N)  should  both  be  defined  more  explicitly. 
Also,  there  are  many  ways  the  systems  can  fail  that  are  not  captured  (eg.,  no  response, 
misordered  input,  didn’t  understand  domain,  produced  garble,  just  paraphrased  in¬ 
put).  Could  also  note  certainty  of  scores.  Not  clear  what  is  meant  by  “demonstrated 
understanding.”  This  was  interpreted  as  “did  it  pretty  much  do  the  right  thing?”  In 
general,  the  criteria  are  positive  and  necessary  -  just  need  to  be  improved. 

IT  #2:  An  improvement  over  last  time,  but  they  are  too  vague  at  times  (e.g.,  saying  the 
system  must  “demonstrate  understanding”  requires  a  lot  of  judgement).  Also,  consider 
having  different  criteria  for  different  kinds  of  NLP  systems. 

IT  #3:  The  scoring  criteria  need  more  explicit  definition.  It  should  be  made  clear  whether 
a  strict  or  conversational  standard  applies.  Consider  demonstrating  proper  application 
of  criteria  with  examples. 

IT  #4:  The  criteria  helped,  but  they  need  to  be  more  clear.  What  is  the  point  of  com¬ 
parison?  Is  comparison  to  what  would  be  expected  of  a  human  or  a  machine?  Also 
consider  making  F  and  N  scores  more  explicit  (e.g.,  word  level,  syntax  level,  semantics, 
and  when  system  just  paraphrased  input). 

5.  Adequacy  of  Training 

IT  #1:  Training  was  important  for  proper  application  of  the  Tool,  but  the  criteria  and 
instructions  should  be  improved  so  training  will  not  be  needed. 

IT  #2:  The  training  was  somewhat  adequate,  but  essential.  Application  of  the  criteria 
takes  too  much  interpretation  and  hence  training  is  needed.  If  the  criteria  can  be 
made  more  precise,  training  would  be  less  needed. 

IT  #3:  The  training  was  somewhat  helpful.  However,  there  was  still  learning  taking  place 
during  Tool  application.  Evaluators  should  be  trained  to  go  back  over  their  answers 
to  look  for  consistency. 

IT  #4:  Training  is  not  very  important,  but  some  instruction  is  required  to  ensure  consistent 
application  of  the  Tool.  Training  embedded  in  the  Tool  which  presents  solved  (i.e., 
scored)  examples  would  be  an  improvement.  This  embedded  training  could  include 
allowing  evaluators  to  practice  with  automated  feedback. 
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6.  Adequacy  of  Structure 


IT  #1:  Some  chapters  are  too  long  and  should  be  broken  down.  Some  of  the  long  chap¬ 
ters  cover  very  small  distinctions.  Often  later  items  prompt  reconsideration  of  earlier 
answers.  Also,  sometimes  failed  structures  passed  in  a  new  context.  Didn’t  have  any 
trouble  remembering  what  had  previously  failed. 

IT  #2:  The  Tool  is  very  exhaustive.  Seems  like  things  are  tested  over  and  over  in  full 
combinatorial  style.  Seems  like  overkill.  Can  the  tool  be  shortened  to  cover  all  the 
material  with  less  repetition? 

IT  #3:  Should  organize  in  a  way  that  will  allow  stopping  when  a  failure  makes  subsequent 
items  untestable  (i.e.,  eliminate  variations  on  failed  features). 

IT  #4:  Didn’t  have  trouble  remembering  what  had  failed,  but  on  one  system  (System  2) 
early  failure  with  a  structure  diinH  mean  that  later  structures  including  this  structure 
would  fail.  I  believe  this  is  because  the  system  looks  for  keywords. 

7.  Improvements  Ideas 

IT  #1:  Make  the  scoring  criteria  more  specific. 

IT  #2:  Covered  under  other  topics  throughout. 

IT  #3:  Should  have  a  scoring  category  called  “response  inadequate”  meaning  that  the 
NLP  system’s  understanding  could  not  be  assessed.  This  was  a  problem  with  System 
2  which  built  database  templates. 

IT  #4:  Some  sections  of  the  Tool  need  to  be  marked  for  “parallel  execution”  since  failed 
structures  may  work  when  used  within  more  complicated  structures  (especially  true 
for  System  2). 

8.  Greatest  Strengths 

IT  #1:  By  applying  the  Tool  to  different  systems  you  will  get  a  sense  of  which  is  best  for 
a  given  application  irrespective  of  the  actual  score  derived  (i.e.,  tool  strength  is  that  it 
forces  one  to  thoroughly  experience  the  NLP  system). 

IT  #2:  Wide  coverage  of  linguistic  phenomena. 

IT  #3:  Exhaustiveness  and  good  examples. 

IT  #4:  Comprehensiveness. 
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9.  Greatest  Weaknesses 


IT  #1:  It  takes  a  lot  of  time  to  apply  the  Tool.  Also  the  nature  of  the  Tool  drains  your 
mind  -  should  limit  application  sessions  to  4  hours. 

IT  #2:  Vagueness  of  criteria.  Dreadful  boredom  in  tool  application  (“this  is  brain-dead 
work  of  the  worst  kind”).  The  boredom  affects  scoring  somehow. 

IT  #3:  Tediousness.  Also  there  were  a  few  bad  examples. 

IT  #4:  The  Tool  is  too  general  to  be  applied  to  all  kinds  of  NLP  systems  -  perhaps  use 
only  for  database  systems. 

10.  Features  That  Should  be  Added 

IT  #1:  Reorganize  larger  chapters  so  they  aren’t  so  tedious.  Also,  you  could  provide  on-line 
error  checking  of  inputs. 

IT  #2:  Provide  a  more  automated  way  of  grabbing  examples  for  use  in  forming  inputs  and 
taking  results  from  the  NLP  system  for  use  in  documenting  scores  in  the  Tool. 

IT  #3:  Provide  examples  that  illustrate  what  responses  would  receive  what  scores.  Provide 
more  automated  ways  of  grabbing  examples  for  use  in  forming  inputs. 

IT  #4:  Automate  decisions  about  what  items  should  be  tested  based  on  earlier  failures.  Add 
on-line  training.  The  Tool  should  be  used  to  define  guidelines  for  further  investigation. 
For  example,  when  an  NLP  system  fails  an  item  the  system  could  aid  in  specifying  the 
cause  of  failure. 

11.  Other  Comments 

IT  #1:  “The  Tool  can  be  very  useful  for  NLP  systems  and  particularly  useful  to  people 
who  are  developing  NLP  system  to  judge  two  well  they  are  accomplishing  the  different 
areas”  The  glossary  did  not  provide  enough  detail.  All  parts  of  each  test  item  are 
equally  important  and  useful. 

IT  #2:  Limit  daily  evaluation  time  to  approximately  two  hours  to  prevent  boredom  from 
affecting  the  results. 

IT  #3:  Consider  completely  automating  the  Tool  using  intelligent  software  to  eliminate 
the  human  element  or,  if  this  is  too  ambitious,  build  a  system  to  aid  the  human  in  the 
evaluation. 

IT  #4:  None. 

<us  GOVERNMENT  PRINTING  OFFICE 
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MISSION 


OF 

ROME  LABORATORY 

Rome  Laboratory  plans  and  executes  an  interdisciplinary  program  in  re¬ 
search,  development,  test,  and  technology  transition  in  support  of  Air 
Force  Command,  Control,  Communications  and  Intelligence  (C3I)  activities 
for  all  Air  Force  platforms.  It  also  executes  selected  acquisition  programs 
in  several  areas  of  expertise.  Technical  and  engineering  support  within 
areas  of  competence  is  provided  to  ESD  Program  Offices  (POs)  and  other 
ESD  elements  to  perform  effective  acquisition  of  C3I  systems.  In  addition, 
Rome  Laboratory's  technology  supports  other  AFSC  Product  Divisions,  the 
Air  Force  user  community,  and  other  DOD  and  non-DOD  agencies.  Rome 
Laboratory  maintains  technical  competence  and  research  programs  in  areas 
including,  but  not  limited  to,  communications,  command  and  control,  battle 
management,  intelligence  information  processing,  computational  sciences 
and  software  producibility,  wide  area  surveillance/sensors,  signal  proces¬ 
sing,  solid  state  sciences,  photonics,  electromagnetic  technology,  super¬ 
conductivity,  and  electronic  reliability/maintainability  and  testability. 


